In an exclusive interview, Lochte talks to PEOPLE about the aftermath of his scandal

Ryan Lochte has been doing a lot of apologizing lately.

“I feel like let down a lot of people,” he told PEOPLE in an exclusive interview on Sunday afternoon. “I feel bad that I’ve let them down.”

Lochte, of course, is talking about the bizarre incident at a Rio gas station in which he and three other swimmers were held at gunpoint.

At first Lochte said it was a robbery, spinning an elaborate tale to NBC’s Billy Bush. Later, he backed off on many of the details after Brazilian police alleged that he had vandalized a gas station bathroom – and that a security guard was simply trying to recoup their losses.

Lochte now tells PEOPLE that he was untruthful when he detailed the incident as a “robbery” in his interview.

“I made that up,” he says. “The part that I made up was saying that it was a robbery. I think the word ‘robbery’ shocked a lot of people, and the story went viral after that. A gun was pulled, and we had to give them cash. If I didn’t say ‘robbery,’ none of this would have happened.”

For more of PEOPLE’s Olympic coverage, pick up our collector’s edition, The Best of the Games, on sale now.

Lochte says he was confused during the incident, and truly thought he was being robbed. “I was going off their body language,” he says. “If we were both speaking the exact same language, we wouldn’t have argued, and things wouldn’t have gotten heated.”

What He Remembers

Lochte is the first to say that his actions were “immature” and “stupid” – and says that he doesn’t recall much of what transpired that night. “I remember some things that happened, but there are a lot of things that I don’t remember,” he says. “There are bits and pieces that I do remember.”

Lochte also says that he was drunk he next morning, when he gave an his interview to Billy Bush.

But Lochte wants to make one thing clear. “I don’t have a drinking problem,” he says. “We hadn’t been able to drink in like months. We hadn’t gone out or partied. But we had just won a gold medal. We were out celebrating. I had one too many.”

Making Amends

Lochte says one of his biggest regrets was getting his fellow swimmers into trouble. James Feigen has been assessed a fine of nearly $11,000. Gunnar Bentz and Jack Conger were not charged, but were pulled off a plane for questioning.

Regarding Feigen’s fine: “I’m going to pay that for him,” says Lochte. “He doesn’t deserve to have to pay that. It was my actions that got us all in trouble.”

RELATED: Ryan Lochte Admits He "Over-Exaggerated" Rio Robbery Story

Lochte says that he has called the swimmers to smooth things over. “I reached out to them and apologized,” he says. “I told them, ‘My immature behavior was stupid, and I’m sorry that I got you guys in this mess.’ They were like, ‘Of course. We just hope that this blows over soon.'”

But before it blows over, Lochte wants to make one thing perfectly clear: he’s not done apologizing.

“I’m sorry if what I did caused any problems in [Brazil],” he says. “I hope that they’ll accept my apology because I am truly sorry. I wish I could change it.”

For much more from Lochte, including his plans for the future, pick up this week’s copy of PEOPLE, available on stands Friday.