Karamo Brown Says His Former 'Dancing with the Stars' Competitor Sean Spicer 'Can't Dance'

"I was robbed, I should have still been there," Karamo Brown admitted on Tuesday's Watch What Happens Live with Andy Cohen of his Dancing with the Stars experience

Karamo Brown may have developed a friendly relationship with his former Dancing with the Stars competitor Sean Spicer, but that doesn’t mean he’s blind to Spicer’s faults — including the fact that the former White House press secretary isn’t the best dancer.

“He can’t dance!” Brown admitted on Tuesday’s Watch What Happens Live with Andy Cohen. He added, “I was robbed, I should have still been there.”

The Queer Eye star, 39, was eliminated from the ABC dancing competition’s 28th season back on Oct. 28, alongside his pro partner Jenna Johnson — despite scoring higher than Spicer that week (and every week). In fact, since the second elimination, Spicer has consistently been given the lowest scores from the judges, though has remained on the show due to fan votes.

“Middle America watches the show and they vote for him,” Brown said on WWHL, noting how President Donald Trump has been supporting Spicer on Twitter. “Our president, who should be doing other stuff, has been tweeting [to] vote for the man!”

Rodin Eckenroth/Getty Images; Shannon Finney/Getty Images

Ultimately, Brown told Cohenthat he had “an amazing time on the show.”

He had previously come under fire for his reaction to Spicer, 48, when both were announced as cast members on the show in August. Rather than outwardly disapprove of the casting, Brown noted at the time that he looked forward to having “respectful conversations” with Spicer — which sparked outrage online (The political aide was widely disparaged after he used the White House press podium to lie to and attack journalists during his tenure; he resigned in 2017.)

Things got so bad that Brown had to temporarily disable his Twitter account.

“I deactivated because they were giving my sons death threats because I was nice to Sean Spicer,” Brown recalled on Tuesday’s WWHL. “There was no friendship [at that point]. I was just saying that, if we’re going to be on the same show, I’m gonna have a respectful conversation with someone who’s different than me.”

“People were like, ‘How dare you? Oh my gosh!’ ” he added. “And I was like, ‘Hey, if you’ve never seen me on television, I’m always gonna be the person to build a bridge.’ ”

Brown said similar things while speaking to reporters, including PEOPLE, after his elimination.

“My goal coming into this was to introduce myself to people who have never seen Queer Eye, to let them know who I am,” he said. “And also, my goal has always been to show how if you can show kindness and lead with love, things can change.”

“Day one I got backlash [for supporting Spicer],” he continued. “I started this show with people telling me, ‘You’re horrible, you’re crazy, you’re stupid.’ Because I showed someone who has a different political view than mine kindness.”

The reality star — who first gained fame on The Real World: Philadelphia and became a political activist, working with the Obama administration on after-school programs for LGBTQ youth — also said he’s “proud” of Spicer.

“Through this process, Sean’s trailer has been next to mine and I have literally every day been planting seeds in his heart,” Brown told reporters. “ ‘When you made this comment, it hurt me as a man. When you made this comment, it hurt me as a person of color. When you made this comment, it hurt me as a gay man.’ And literally, I could see each week those seeds … blossoming.”

“Sean is literally someone who I would’ve never thought I could be friends with, and I’m going to walk away from here calling him a friend,” Brown said. “I’m going to continue the conversation that we’ve had outside of this. And I don’t think Sean’s going to make it to the end [of the competition], but I’m proud of him. I really am proud of him, because he’s had fun every week and I think he’s exceptional.”

Spicer echoed Brown’s sentiments during his own interview with reporters that night.

“It just speaks to what a great person he is, that he’s trying to bridge divides,” Spicer said of his “friend” Brown supporting him from the get-go. “I think that it’s been great getting to know him, getting to discuss things with him. And so it’s tough to watch somebody that you’ve really got close to go home when you know how hard they’ve worked and how much progress they’ve made.”

Eric McCandless/ABC

Karamo Brown on Dancing with the Stars. Jenna Johnson

On Monday, Spicer opened up to PEOPLE about his part in the competition.

“I knew what my scores were and I knew how the rest of the cast did,” Spicer said. “To be called out first was just a massive relief. For me, this week was about bouncing back from last week and showing Lindsay [Arnold] that all of her hard work had paid off.”

“I know how hard everyone else is working too,” he continued. “You feel bad because you’ve seen their progression. It’s tough. It’s this mixed emotion we’ve had for the past few weeks, where we’re watching someone you’re close to and watch them go when your scores weren’t as high.”

Meanwhile, judge Carrie Ann Inaba told PEOPLE she was nearly at a loss for words seeing him beat The Office star Kate Flannery, who was sent home alongside her pro dancer partner Pasha Pashkov.

“Tonight was shocking, especially when you see somebody with the caliber of dancing who got a 30, Ally today, with Kate out there,” she said. “We’ve seen already Karamo went home last week, and we’re seeing good dancers go home. And this season we’ve upped our game as far as judges, and we’re really going after the technique in effort to make sure that the best dancer wins.”

“The competition is fierce this season,” she continued. “I feel like everybody’s elevated to that level. But people are voting the way they want to. And they have a right to, but I would like to beg the public to remember that this is a dance competition. Of course, vote with your heart, but also take a moment to consider everybody else in the competition, and are you voting for the right person? I’m not sure.”

Dancing with the Stars airs Mondays (8 p.m. ET) on ABC.