The surprise guest was in attendance at Shep Rose's birthday party

Austen Kroll and Madison LeCroy are working out their issues after a surprise guest showed up at Shep Rose's birthday party. 

In PEOPLE's exclusive sneak peek at Thursday's episode of Southern Charm, Austen, 33, admits that "it hurt" when a guy Madison, 31, once slept with came to the party.

Though she and Austen were broken up when the one night stand happened, his appearance at the party led Madison to suspect that Shep, 40, had set the whole thing up to create conflict. (The two famously didn't get along last season.) Shep, however, insisted it was just a coincidence. 

"I think it kind of sucks that obviously the Peter thing came out," Madison tells Austen the next morning. "But you can't let what people say affect how we are right now, because right now we've been doing so well."

Southern Charm
Credit: Bravo

"I mean, yeah, it hurts," Austen replies.  

"Listen," Madison continues. "People are going to continue to try to break this relationship up and it makes me sick."

Though Austen acknowledges that he can't hold the one night stand against Madison because they weren't together at the time, he's worried that Shep is trying to sabotage their relationship.

"It just sucks because you and I made a choice to come back to each other and let it all go," he says. "I mean, I can't crucify you for something that happened when we weren't officially together. I just don't want Shep creating a wedge [between us] — that can't happen."

Austin Kroll and Madison LeCroy
Austen Kroll and Madison LeCroy
| Credit: Austin Kroll/Instagram

"I just want all this to stop," Madison says before Austen adds of Shep, "I hate that he has knowledge that I don't."

"Don't you understand that when these things come out, they're like, testing to see how strong we are?" Madison tells him. "We've been through enough s--- that you can't break it, and I don't want to be with anybody but you."

RELATED VIDEO: Austen Kroll on Shep's Girlfriend, 3 Cast Members' Exits and Pitching Friends for Southern Charm

Austen and Madison have had a whirlwind relationship, often breaking up and then coming back to each other. Last November, he opened up about the on-off romance during the Southern Charm cast panel at BravoCon, saying the two were "doing well" at the time.

"This week I'm doing well with Madison," he said last year. "Maybe next week that will change, but I'll let y'all know."

"We've tried to go our separate ways," he added. "Madison has been a person for me for a long time and it's really tough to walk away from something like that. She's a hell of a woman and I find it hard to walk away from."

Southern Charm airs Thursdays (9 p.m. ET) on Bravo.