Michael Phelps Tells PEOPLE He Would 'Love' a Daughter: It's Time for the 'Next Chapter of My Life'

"It's time for me to start the next chapter of my life," the Olympic swimmer told PEOPLE in Rio Tuesday

Matt Slocum/AP

Michael Phelps may be retiring from his Olympics career, but he has plenty to look forward to in parenting 3-month-old son Boomer Robert — and a possible new addition.

“We’ve talked about more [children]. We don’t really know how many more we want,” the 31-year-old swimmer said Tuesday in a PEOPLE live chat from Rio. “I think we’re gonna have some fun with Boomer before we make that next choice of trying for another one.

“We would love a girl. Love, love, love a girl,” he adds. “We wanted a boy first, just in case we did have other children, [so] he could be the protector of his younger sibling or siblings.

“So we started off on a perfect note: We got a boy. We got a beautiful, healthy baby boy, and we’re looking to continue to grow our family.”

Want all the latest pregnancy and birth announcements, plus celebrity mom blogs? Click here to get those and more in the PEOPLE Babies newsletter.

This embed is invalid

While in Rio, where Phelps recently completed his last Olympic race ever and won a total of six medals, including five gold, the new dad announced his plans to retire from the sport.

“It’s time for me to start the next chapter of my life, and I’m looking forward to that now,” he says.

“I have a beautiful fiancée, a beautiful baby boy [whom] I want to spend as much time with as I can,” he adds. “I’ve been able to accomplish everything I could ever dream of [with professional swimming], and it’s time to move on.”

Despite his rigorous training and competition schedules in Rio, Phelps has managed to squeeze in quality moments with his son — including time spent on diaper duty.

“I changed a couple diapers since I [came here] with my family [to] Rio, and it’s just awesome watching him smile,” says the new dad, whose fiancée Nicole Johnson was teary-eyed from the stands as Phelps won his last gold in Rio.

“Watching the facial expressions he has [is] the greatest, because he’s laughing now, he’s making noises. [It] sounds like he’s saying ‘Hi,’ but he’s not,” Phelps continues. “He’s just very energetic. He’s locking onto things and staring at things now, so you really do see him growing — not just physically growing, but you can tell he’s getting older. And it’s just a cool feeling watching him grow up.

“I’m looking forward to not missing a day after this,” he adds.

If Boomer wants to get into swimming, his dad will be supportive — but he won’t push it.

“If he wants to get into swimming, that’s fine. If he doesn’t, I’m fine with that too,” Phelps says. “I was never pushed as a kid to put more time into the pool, or play other sports. My mom kinda let me make the call and choose what I was gonna do.

RELATED VIDEO: Check out Boomer Phelps’s Adorable Olympic Outfits!

“And I think that’s something that’s helped me stay in this sport for so long and still be in love with the sport,” he admits. “So hopefully we’ll be able to do that with Boomer, and hopefully he finds a sport or he finds something [else] that he loves, and he just goes for it.

“I’ll teach him,” the proud dad continues of how he’ll be involved in his son’s swimming if that’s the path he chooses to take. “I’m gonna take him in the pool when I get home.”

And it turns out Phelps and Johnson — who plan to marry in a “destination, small wedding” around the end of the year — are both in luck when it comes to their son’s looks.

“I’m starting to see a mixture of myself and Nicole. I see more of her right now, [but] he is growing a long torso, so that’s something that I have that he hopefully will get,” Phelps says, joking, “Hopefully he grows some legs because I have very short, stubby legs.”

— Jen Juneau with reporting by Steve Helling