"I will not be in the pool competing," Michael Phelps said of the 2020 Olympic Games in Tokyo

Michael Phelps fooled the world once when he said his Olympic career was over following the 2012 games in London. But after a triumphant surprise comeback at the Rio games (including five gold medals), he insists he’s done for good now.

“Done, done, done. This time I mean it,” the 31-year-old told Matt Lauer. “I wanted to come back and finish my career how I wanted. And this was the cherry on top of the cake that I wanted.”

And what of rival and pal Ryan Lochte‘s “guarantee” that both swimmers would meet again in Tokyo in 2020?

“He might be there, but I won’t,” Phelps told Lauer with a smirk on Monday, appearing annoyed by his teammate’s comment. “I’ll be there watching, cheering everybody on. But I will not be in the pool competing.”

Later, when Lauer asked a slew of U.S. Olympians whether they thought Phelps was really done this time, nobody raised their hand – but gold medal swimmer Lilly King gave the proposition a wag of her famous finger.

“I don’t know, we hope not,” King said of Phelps’ retirement plans.

Now, with 28 Olympic medals – 23 of them gold – and the title of the most decorated Olympian of all time, Phelps says he’s done for good.

But the athlete said the future of swimming is in good hands with Katie Ledecky.

“We are in great shape,” he said. “Katie’s doing her thing and she’s breaking records every time she gets in the pool.”

“It’s been an honor for me to be a part of the team with her and with some of the younger guys that we have coming up in the sport … I’ve told a bunch of them ‘I won’t be far away.'”

So, what’s next for Phelps?

“This is the part of my life where I get to start this whole new chapter,” he said, noting that he’d like to “continue building a family” with his fiancée Nicole Johnson. The two welcomed son Boomer in May.