Nancy Kerrigan Didn't Stop Trying for Baby No. 2 Despite 6 Miscarriages: 'I Wasn't Ready to Stop Trying'

"Jerry asked me if I was sure I wanted to keep going. It was hard for him to see me hurting," Nancy Kerrigan says of her husband's concern through her miscarriages

Nancy Kerrigan knew early on that she wanted a full house of children.

“I always thought I’d have three kids by the time I was 30,” the two-time Olympic medalist in figure skating, 47, says in an current issue of PEOPLE.

And after she wed her agent Jerry Solomon in 1995, Kerrigan quickly became pregnant and welcomed her son Matthew in 1996.

“He went on tour with us for months and it was great,” says Kerrigan, who had transitioned to professional skating after her silver medal win at the 1994 Winter Olympics.

Damian Strohmeyer /Sports Illustrated/Getty

But as easy as her first pregnancy had been, it would be eight years before she would become a mom for the second time.

“The first time that you go in and they tell you, ‘Oh there’s no heartbeat,’ it’s devastating,” Kerrigan says of suffering “at least” six miscarriages in the years after giving birth to Matthew.

“I felt like a failure,” says Kerrigan, whose doctors were never able to determine why she kept miscarrying.

Still, Kerrigan, who was raised with two older brothers, would not be deterred from having the family she’d always envisioned.

Photograph by Brian Doben

“Jerry asked me if I was sure I wanted to keep going. It was hard for him to see me hurting,” she says. “But I wasn’t ready to stop trying.”

After undergoing in vitro fertilization treatment (IVF), Kerrigan finally welcomed Brian in 2005.

“I think about it now and remember we couldn’t come up with a name for Brian,” says Kerrigan. “I wonder if we probably were afraid to come up with a name because that makes you close and we could lose him.”

• For more from Kerrigan — including how she coped with the media attention after she was clubbed in the knee just weeks before the 1994 Winter Olympics — pick up the upcoming issue of PEOPLE, on newsstands Friday

But through the fear, Kerrigan began the IVF treatment again to have her third child.

“There were two eggs left and they said, ‘Do you want both?’ And we were, like, ‘Oh.’ Sometimes people get twins and that would have been okay before Brian, but we already had two now so we said no,” she says. “They said, ‘This one looks strong!’ But then it wasn’t strong and they said it didn’t work. Then they said ‘There’s only this weak one left,’ which is funny because our daughter Nicole is the complete opposite of weak.”

“Now we’re outnumbered,” laughs Kerrigan, who gave birth to Nicole in 2008.

From Coinage: The Cost of Dancing With the Stars’ Mirrorball Trophy

Courtesy Nancy Kerrigan

Kerrigan has done commentary for televised figure skating competitions and occasionally works as a skate instructor, but “my job is being with my kids after school and asking them how their day was,” she says.

As the children grow up —Matthew is thriving as a costume designer, Brian is a competitive gymnast and Nicole is dedicated to ballet— however, Kerrigan is open to new adventures like DWTS.

“We’ve always watched DWTS as a family,” says Kerrigan. “So the kids are so excited to get to watch me on it now.”

Dancing with the Stars airs Mondays (8 p.m. ET) on ABC.