During the recent Southern Charm reunion, Madison LeCroy was accused of sleeping with a married ex-MLB player, which she denied

Southern Charm's Madison LeCroy is speaking out amid speculation that she had an affair with Alex Rodriguez.

Following Thursday's episode of the Bravo reality series, when LeCroy was accused by some of her castmates of sleeping with unnamed, married ex-MLB players, rumors swirled that one of the men in question might have been Rodriguez.

Now, LeCroy is alleging that she and the longtime professional baseball player have communicated over the phone, but is denying that they ever had a physical relationship.

A source who knows Rodriguez, 45, tells PEOPLE: "It's a B.S. story. Alex has never met her."

In an interview with Page Six on Wednesday, LeCroy claimed that the pair have "spoken on the phone" — in calls that she said were "innocent" — but said that they have "never met up" and "never been physical … never had any kind of anything. Just an acquaintance."

Madison LeCroy, alex rodriguez
Madison LeCroy (L); Alex Rodriguez
| Credit: getty images (2)

She went on to allege that Rodriguez — who is engaged to Jennifer Lopez — has "never physically cheated on his fiancée with me," claiming that she has spoken with him "randomly, but not consistent."

"All this stuff was a year ago [but] it's being aired now," said LeCroy, referring to the time between when the messages allegedly took place and when Southern Charm began airing.

Madison LeCroy
Madison LeCroy
| Credit: Madison LeCroy/Instagram

During last week's reunion of Southern Charm, Craig Conover accused LeCroy of sleeping with other men while she and then-boyfriend Austen Kroll were still working out the status of their relationship.

"You were flying around the country sleeping with men, married men!" Conover claimed. "Ex-MLB players. That's what you were doing during quarantine. You know why Austen was at my house? 'Cause you weren't talking to him! You flew to Miami to f--- an ex-MLB player."

"That's the godd--- truth," Kroll added.

LeCroy adamantly denied the allegation, calling the suggestion that she flew to Miami "fake."

"Give me a name!" she fired back. "Put me on a lie detector test. I never flew to Miami. Where's the record of that? That's fake."

The two men went on to claim that LeCroy openly flaunted messages between herself and the former athlete. While the former baseball player's name was bleeped out, host Andy Cohen clarified that the man in question was a "very famous, married ex-MLB player."

Speaking to Page Six, LeCroy said that the mystery name was in fact Rodriguez's, but she "told them they couldn't use that."

"I have tried to be as quiet as possible. I don't want anything bad for his family, or for mine," LeCroy said on Wednesday. "We are definitely innocent in this."

Her denial comes more than a week after LeCroy appeared to suggest that she had a brief fling with Kristin Cavallari's ex, Jay Cutler.

In January, LeCroy shared screenshots of her alleged texts with the former Chicago Bears quarterback on social media and referencing Cavallari in an Instagram Live video.

Madison Lecroy and Jay Cutler
Madison LeCroy and Jay Cutler
| Credit: Madison Lecroy/Instagram

Within the messages, Cutler, 37, appeared to ask LeCroy to "hang out." She also shared a selfie of herself and Cutler on her Instagram Story and wrote, "Too bad it didn't work out." (A rep for Cutler did not respond to PEOPLE's previous request for comment.)

A source previously told PEOPLE, "Jay reached out to Madison and they spent time together. She flew out to see him after filming the Southern Charm reunion last month."