"A lot of people have done a lot worse" Gaines tells PEOPLE
Credit: Getty (2)

The hits keep coming for Ryan Lochte, but at least one former Olympian is coming to his defense.

“He’s a really good guy,” NBC swimming commentator and three-time Olympic gold medalist Rowdy Gaines tells PEOPLE, adding: “Did he make an amazingly stupid decision? Absolutely. Is it forgivable? I hope so. I hope we live in a country where something like this can be forgiven.

“A lot of people have done a lot worse. It doesn’t make it right, it just makes it forgivable.”

Gaines, 57, says he’s been heartbroken by Lochte’s alleged fabrication of an armed robbery in Rio, during the Olympic Games, involving teammates Gunnar Bentz, Jack Conger and James Feigen – because that’s not the athlete Gaines knows.

As proof, he recalled how the 32-year-old swimming champion recently spent hours signing autographs for young swimmers.

“We had a swim meet here at the YMCA in Orlando a few months ago with 900 kids and he signed 900 autographs, stayed for four hours, until 11 p.m. at night,” Gaines says.

The veteran swimmer says he was most shocked by Lochte’s continued insistence on the robbery story, even as contradictory evidence mounted.

“It breaks my heart that Ryan kept going,” Gaines tells PEOPLE. “After it first came out he should have kept his mouth shut, not gone on the Today show and kept talking about it.”

For more of PEOPLE’s Olympic coverage, pick up our collector’s edition, The Best of the Games, on sale now.

Lochte took to Instagram on Friday to apologize about the controversy and accept responsibility for the incident, which authorities now say was not a robbery but rather a heated, if confusing, confrontation with armed gas station security over alleged vandalism.

But Gaines says Lochte’s statement should have come much sooner: “Once he got his mind right he should have realized, ‘Holy crap, this isn’t right and I have to make it right now.’ ”

Instead, on Thursday, the U.S. Olympic Committee CEO Scott Blackmun issued an unprecedented apology to the entire nation of Brazil on behalf of Lochte and his three teammates. Blackmun said the athletes’ behavior was “not acceptable” and not representative of Team U.S.A.’s values.

Road to Rio: How Did Team USA Train for the Olympics?

A USOC source told PEOPLE Friday that if the committee had its way, Lochte and Feigen may not return for another Olympic Games.

“The thinking is that no amount of gold medals is worth this,” the source told PEOPLE. “They totally took over the entire Olympics with foolishness.”

Despite these statements, Gaines says a ban isn’t warranted. The 45 members of the U.S. swim team are “hurt and upset,” but the swimming world is a “small family,” he says.

And in order to be forgiven by them, Lochte and the other swimmers involved will need to own up to their mistakes and repent, he says. Not just for the lies, but for drawing attention away from Olympic excellence.

“If they do this the right way, then we, the swimming community, will forgive them,” Gaines says. “It’s forgivable.”