An expert tells PEOPLE Phelps' troubled past could affect his future earnings now that he's said goodbye to competitive swimming

By Char Adams
August 19, 2016 04:20 PM

Michael Phelps‘ historic success as an athlete has brought him fame and fortune – but as his last Olympic Games wrap up, in Rio, Phelps faces the risk of greatly diminished earnings out of the limelight, one business expert tells PEOPLE. Even with 23 gold medals to his name.

So what will happen to his fortune now that he’s retired from competitive swimming? Dr. Patrick Rishe, the director of the sports business program at Washington University in St. Louis, details the problems and possibilities:

Phelps, 31, has an estimated net worth of $50-$55 million, Rishe tells PEOPLE. But that number pales in comparison to soccer star Cristiano Ronaldo, with a $50 million yearly salary, or tennis champion Serena Williams, who has a net worth of $150 million.

Michael Phelps

This, Rishe says, is due to the fact that, as an Olympian, Phelps is in the public eye less frequently than other athletic stars, such as basketball players or golfers.

For more of PEOPLE’s Olympic coverage, pick up our collector’s edition, The Best of the Games, on sale now.

“As such, companies are not as willing to make huge investments in less visible athletes,” says Rishe, founder and CEO of SportsImpacts. “Because Phelps is not a shining new star, there’s no ‘wow’ factor like there is with Simone Biles.”

Rishe adds: “Because [Phelps] has likely swum in his last competitive meet, companies wonder how valuable a corporate endorser he will be if he is not regularly in the public eye.”

He’s hardly fallen on hard times, though: Phelps is likely to have collected a total of $125,000 in Rio, for each of his gold medal wins, as the United States Olympic Committee awards Olympians $25,000 for each gold medal. Plus he earned a silver medal, for which the USOC awards $15,000.

His endorsement deals throughout his career have also brought in an estimated $94 million, Fox Business reports.

Phelps has a slew of medals and swim meets behind him, along with two books and his newly released Phelpsmojis. His endorsement portfolio boasts names like Under Armour, Omega and a slew of smaller companies, according to CNBC.

Michael Phelps

But, Rishe says, his past mistakes could cost him future deals.

“While I appreciate Phelps’ greatness … companies were – and are still – turned off by Phelps’ past peccadilloes, including two DUIs, the infamous ‘bong photo,’ ” Rishe tells PEOPLE. “And other past activities which don’t leave a corporate executive with a warm fuzzy [feeling] about embracing Phelps as their new corporate pitchman, no matter how many gold medals he wins.”

Phelps lost endorsement deals after the scandals and didn’t even make it on to Forbes’ August list of 14 Olympians who earned at least $18 million in a year’s time. (NBA star Kevin Durant came out on top with $56.2 million.)

Road to Rio: How Moms Support Olympic Hopefuls

However, Forbes noted that even the highest-ranked Olympians have earned less than $15,000 a year from all sources of income, including sponsorships, prize money and grants.

Rishe estimates that Rio stars such as Biles and Katie Ledecky will see up to $8 million in the next two years thanks to their newfound fame – “then it might die down as the memories of their accomplishments fade from the public’s mind.”

So the best bet for Phelps, in more ways than one, is to focus on his relationship with Under Armour, Rishe says.

“It’s a loyal partnership,” he tells PEOPLE. “I think it’s the Under Armour relationship which will produce his greatest opportunity to profit from endorsements long-term, as well as continue to rehabilitate his image.”