In a PEOPLE exclusive teaser for Sunday's episode, Garcelle Beauvais, Teddi Mellencamp Arroyave, Dorit Kemsley, Lisa Rinna and Kyle Richards battle it out against Cohen and his team members John Hill, Amy Phillips, Bevy Smith and Jeff Lewis. (RHOBH's Erika Girardi and Denise Richards did not participate.)
"They told us that the answers were really easy," Cohen, 52, says backstage, to which Smith responds: "And they weren't."
"It was a little more high-concept than I think any of us expected," Cohen explains.
As for Richards' time on the show? "It was just really fun playing opposite our boss and being able to take him and his team on," she says, adding, "That was fun."
"I thought the Beverly Hills Housewives were great," says Cohen. "I knew they were going to be great."
However, Lewis believes the ladies were a bit tough.
"There was a little psychological warfare," Lewis says. "There was almost bullying. I was hoping they'd be nicer in real life — not so much. They're bitches, basically."
Still, it seems the upcoming episode is filled with lots of laughs and friendly competition.
"This reaction from all of them!" host Steve Harvey says to the RHOBH as they giggle uncontrollably after answering a survey question.
"It was kind of strange to see Andy not be the best at something because he's good at everything," Kemsley, 43, says, to which Rinna, 56, jokingly quips, "Your storyline just got cut way down!"
"We were trying to be really dumb and I think we succeeded," Cohen jokes.
Reflecting on the experience, Richards, 51, says it was "very exciting" as it was something she watched as a little girl.
Celebrity Family Feud airs Sundays (8 p.m. ET) on ABC.