Thomas Ravenel's Southern Charm costars have all but cut ties with him since filming on season 5 wrapped, a source tells PEOPLE

By Dave Quinn
August 08, 2018 04:51 PM
Thomas Ravenel
Rodolfo Martinez/Bravo

Thomas Ravenel‘s Southern Charm costars have all but cut ties with him since production on season 5 wrapped, a source tells PEOPLE.

An insider close to the cast says the 55-year-old reality star has been ostracized from his circle of friends over the past few weeks, with former friends like Shep Rose, Cameran Eubanks, Craig Conover, Austen Kroll, Chelsea Meissner and Naomie Olindo putting their focuses elsewhere.

Their reasoning? “Eye-opening behavior” from Ravenel this year, the source says, including his relationship with girlfriend Ashley Jacobs; attacks he’s made against Kathryn Dennis — his ex-girlfriend and the mother of his two children — and allegations of sexual assault brought against him by the two’s former nanny, Dawn.

“He doesn’t have friendships with the [main] cast anymore — everyone is done,” the insider explains. “They’re disgusted with him and the way he and Ashley treated Kathryn this season. His eye-opening behavior and the drama surrounding those allegations caused a huge wedge in his friendships.”

“There’s too much dishonesty there, too much manipulation,” the source continues. “They’ve all just been keeping their distance.”

WATCH: Southern Charm’s Thomas Ravenel Accused of Sexual Assault, Allegedly Paid Accuser $200,000

Representatives for Bravo had no comment, but the source adds that only Whitney Sudler-Smith and J.D. Madison — who recur on the Bravo reality show —have remained in contact with Ravenel, as has Dennis. The two share 50/50 custody of their daughter Kensie, 4, and son Saint, 2½.

Dennis was surrounded by some of her Southern Charm cast mates over the weekend, when Rose, pal Danni Baird, and former-foe-turned-friend Patricia Altschul — Sudler-Smith’s mother — helped throw her a surprise 27th birthday party.

“Best birthday that I have ever had… and also the only surprise party anyone has ever held for me!” Dennis tweeted. “Thank y’all for making my day so special!”

Ravenel was not invited to the party, the source says — though it appears unlikely he would have even attended.

“Thomas been like a hermit, staying close to home at his Brookland Plantation,” reveals the source. “He’s completely distraught over the allegations and what’s happened with the show.”

“He’s bounced back from scandal before, but this is much harder now that he has kids,” the source continues.

Thomas Ravenel
Rodolfo Martinez/Bravo/NBCU Photo Bank via Getty

Ravenel has had a history of trouble with the law, dating back to 2007 when — as South Carolina’s treasurer — he was indicted on federal cocaine distribution charges. He was sentenced and served 10 months in jail, and went to rehab as well.

Back in May, news broke that Ravenel was being investigated by South Carolina police for the alleged sexual assault of his nanny. Real estate agent Ashley Perkins has also accused him of assaulting her mom Debbie Holloway Perkins after they met on Tinder in October 2015. Ravenel has denied the accusations through his attorney.

Since then, Bravo and Haymaker (the production company behind Southern Charm) have launched an investigation into the claims, saying they “take allegations like these very seriously.”

“Once all the information is gathered and carefully reviewed, appropriate action will be taken,” they said.

No update into the investigation has been provided yet, but Ravenel did skip the season 5 Southern Charm reunion, which taped in June.

RELATED VIDEO: Southern Charm Kathryn Dennis on ‘Natural Chemistry’ with Thomas Ravenel: ‘There’s Common Ground from the Past’

“I have seen a new Thomas,” she explained. “I mean, it’s humbling, of course, and now he’s [like] ‘Let’s stay in, let’s watch movies, let’s cook dinner, let’s just stay under the radar right now.’ And I’ve just seen a much softer, more sensitive, kinder, more supportive boyfriend. You know, everyone wants to be with you when you’re on top. Who’s with you when you’re down there? And I’ve told him, I’ve just said, ‘I’m not leaving.’ ”

But the rest of the cast wasn’t convinced by her insistence that she and Ravenel are in a healthy relationship and alleged Jacobs was being manipulated by Ravenel in the same way he once controlled her enemy, Dennis.

“You’re exactly where Kathryn was,” said Conover, with Eubanks adding, “You’re lying to yourself, Ashley … What does it say about a man who talks poorly about the mother of his children? … He’s been a not nice person.”

“I feel bad for you,” Dennis said. “I’m watching you be me years ago. I’m watching it unfold right before me.”