"It was intense. I thought, 'Oh man do I have to fight this guy? I don't know how to fight,' " Rose tells PEOPLE

By Colleen Kratofil
Updated June 13, 2016 09:00 PM
Jeffrey Mayer/WireImage

In this week’s episode of Southern Charm things got downright volatile during Thomas Ravenel‘s dinner party.

The 53-year-old playboy invited the whole gang (Kathryn Dennis included!) to celebrate the birth of his son, St. Julien Rembert Ravenel, but before the main course was even served, insults were flying and guests were fleeing on golf carts!

PEOPLE caught up with Shep Rose, one of the show’s voices of reasons, to understand what exactly caused Ravenel’s party to completely unravel.

According to Rose, it all started with Ravenel’s toast to kick off the night. “We hadn’t even gotten our salad and Thomas stands up to make these toasts and, man, things got sideways real quick,” Rose tells PEOPLE. “He kind of went around the room and instead of being sort of playful and poking fun, things kind of got out of control and he got personal and a little bit more mean than he should have.”

And he has a theory as to why Ravenel started with all the jokes turned insaults in the first place. “Thomas in the first season did a scene at his dinner table. It was funny. He was sort of critical, but he was lighthearted and everybody sort of rolled their eyes,” Rose explains. “And I think he wanted to replicate that. But this time for whatever reason, maybe he had one too many, or a stiffer bourbon and things snowballed which they tend to do sometimes with Thomas. When it goes bad, it goes bad fast.”

But it seemingly came out of nowhere that Rose would have tensions with Ravenel – just the episode before they were toasting his newborn son together.

“What bothered me was that Thomas told Landon [Clements] she’s been ‘quite catty’ towards Kathryn. And I thought to myself, ‘Wait a minute, it takes two to tango here, Thomas.’ Whatever relationship you have with Landon that bothers Kathryn, it’s a two-way street. They’re friends and they hang out and it really bothers Kathryn. So there’s no way you can point the finger at Landon and say, ‘I’m mad at you because you’re my friend.’ That is such nonsense. I thought to myself, ‘That’s illogical and I’m going to call him out.’ That was what set me off.”

And when Ravenel’s insults hit Cameran Eubanks, Rose wasn’t having any of it. “I stuck up for my friends and I thought he was in the wrong, so he turned on me and got on my case. You can say anything to me, nothing bothers me. I don’t take things personally, but I mean I’m sure not going to back down.”

Things got so heated, Rose thought it could get physical. “It was intense. I thought, ‘Oh man do I have to fight this guy? I don’t know how to fight.’ But I certainly wasn’t going to back down; it was crazy.”

In the end, Rose does see the silver lining of the situation. “I mean he didn’t punch me. That would have sucked. So as long as they’re just words I think everyone can be okay.”

But he does still have beef (so to speak) over what happened that night, “I was hungry, man! In the back of my head I was like, there’s a nice filet mignon in the kitchen. I want that, damn it!”

Southern Charm airs Mondays (9 p.m. ET) on Bravo.