"If you want to go to a higher level, you can't really do a lot of different things," Wieber shares of Jordyn's need to specialize early

The women on the latest edition of PEOPLE’s Mom Talk discuss that when their kids were very young, they wanted to make sure they were well-rounded in the world of athletics.

But at some point, they realized that to be top in one sport, they would have to steer them in that direction. The trouble, though, was figuring out when that should happen.

“When my kids were little, it seemed like we’d let them get involved in a number of different sports,” says Rita Wieber, whose daughter Jordyn Wieber is a former Olympian who helped her team win gold at the 2012 Summer Olympics in London.

“And then Jordyn had to settle into just gymnastics at a certain point,” Rita continues of her daughter, who recently turned 21. “Because if you want to go to a higher level, you can’t really do a lot of different things.”

For more of PEOPLE‘s Olympic coverage, pick up our collector’s edition, The Best of the Games, on sale now.

Jordan Wieber Specializing Gymnastics Early

Courtesy Rita Wieber

Though the Wieber family obviously found a great balance in figuring out the right age for Jordyn to pursue strictly gymnastics, Rita admits it wasn’t easy figuring out what that timeframe should be.

“At what point do you decide, ‘They need to spend all of their time doing this particular sport because they have an incredible amount of talent?’ ” she asks.

“And is it different for each sport?” mother of four Lindsay Davenport adds in agreement.

“I only played tennis from [age] 6 on,” continues the former Olympian. “And so for most tennis players, that is kind of the routine. It is a specialized sport.”

Jen Juneau