'Southern Charm' Cast Is 'Relieved' Thomas Ravenel Is Off the Show, Source Says: 'He's Toxic'

Thomas Ravenel won't appear as a cast member on Southern Charm next season in the wake of his arrest on Tuesday on charges of assault and battery

Photo: Rodolfo Martinez/Bravo

Thomas Ravenel won’t appear as a cast member on Southern Charm next season in the wake of his arrest on Tuesday on charges of assault and battery — and his costars support the network’s decision.

A source close to the cast at the hit Bravo show tells PEOPLE that they’re looking forward to returning for season 6 without Ravenel.

“They’re relieved not to have to deal with Thomas on the show anymore,” the source says. “Thomas was a friend at first, but that changed over the years as layers and layers of his lies and manipulation were exposed. What he did to Kathryn was horrible, but the allegations from his nanny became the last straw.”

“It disgusted so many of them on the show, especially [the female cast members],” the source continued. “They’ve all been keeping their distance since. He’s toxic.”

The source added: “They’re looking forward to start filming again, and happy to be finally be free of that.”

Representatives for Bravo did not respond to PEOPLE’s request for comment. Ravenel’s lawyer has also not returned PEOPLE’s request for comment.

Rodolfo Martinez/Bravo/NBCU Photo Bank via Getty

Ravenel was arrested Tuesday and charged with assault and battery in the second degree, Charleston County jail records show.

Also on Tuesday, Bravo exclusively confirmed to PEOPLE that Ravenel will not be returning to the reality series as a cast member. (The show’s season 5 stars were Shep Rose, Craig Conover, Cameran Eubanks, Kathryn Calhoun Dennis, Austen Kroll and Chelsea Meissner).

RELATED VIDEO: Southern Charm’s Thomas Ravenel Accused of Sexual Assault, Allegedly Paid Accuser $200,000

The network’s decision comes after Ravenel suggested he was quitting in August, tweeting “I’m not doing the show anymore. In the contract they have the right to fictionalize your story. They took advantage of me. I decided I’ve got too much to lose and informed them I’m not coming back.”

Rebecca Miller/Bravo/NBCU Photo Bank via Getty

Sources has previously told PEOPLE that Ravenel’s costars were “disgusted” with him.

“He doesn’t have friendships with the [main] cast anymore — everyone is done,” the source explained. “They’re disgusted with him and the way he and Ashley treated Kathryn this season. His eye-opening behavior and the drama surrounding those allegations caused a huge wedge in his friendships.”

“There’s too much dishonesty there, too much manipulation,” the source continued. “They’ve all just been keeping their distance.”

Bravo

Police began investigating the disgraced politician in May after a woman reported him for sexual assault. A former nanny to his children named Dawn claimed to PEOPLE that she is that woman and alleged that, while his daughter was sleeping nearby, he “corralled” her into the master bedroom, dropped his pants, blocked the door and “proceeded to rip my clothes off.” Without going into further detail, Dawn said she eventually got away.

A source told PEOPLE that Ravenel’s arrest is connected to her allegations.

Ravenel denied Dawn’s allegations through his lawyer, as well as a separate sexual assault accusation from a woman he met on Tinder in 2015.

“My client enjoys a certain degree of fame, which has unfortunately made him a vulnerable target for such claims,” Ravenel’s attorney, Richard P. Terbrusch, told PEOPLE in a statement after the allegations. “The person this woman describes is simply not the man that I know. My client is a loving father, successful businessman, and upright member of the community. He is appalled and hurt by these allegations — and is committed to defending his reputation in the appropriate legal forum.”