Harry Hamlin also refers to the cast of the Bravo series as "desperate housewives"

Credit: Angela Weiss/Getty

Harry Hamlin supports wife Lisa Rinna, but he still likes to tease her about her Bravo career.

“I call them desperate housewives around the house,” Hamlin told Ben Lyons during a IMDb Asks appearance at the Sundance Film Festival on Sunday. “I say ‘Are you doing Desperate Housewives tomorrow?’ ”

The 64-year-old joked that he threatened to call a divorce lawyer when Rinna initially floated the idea of joining The Real Housewives of Beverly Hills.

“She came to me and she said that they had asked her to do it, and I said, ‘If you do it I’m going to call my divorce lawyer,’ ” Hamlin said. ” ‘I happen to have a divorce lawyer in my phone. And as long as I don’t have to be on it, you can do it.’ ”

(For the record, the actor has made brief appearances on the show, even filming a dinner party celebrating his beloved wife’s birthday this season.)

The couple has been married since 1997 and share two children, Delilah, 17, and Amelia, 14. Both girls have appeared on the show, which the actress joined during its fifth season in 2014.

VIDEO: Kyle Richards Dishes on the Drama-Filled Season 6 of The Real Housewives of Beverly Hills

Hamlin isn’t exactly a stranger to reality television, however. Back in 2006, the actor competed on the third season of Dancing with the Stars.

“That is the hardest thing I have ever done in my life,” he said Sunday of the experience. ” I knew how difficult it was because Lisa had done it the season before and it changed her life. It was the kind of thing where you have to rise to this occasion to do this thing that you cannot do.”

The Mad Men actor said he “only lasted for a couple of weeks” because it was “impossible.”

“But, nevertheless, just the experience of doing those couple of weeks changed my life,” he explained.

Hamlin also discussed nabbing the coveted title of PEOPLE’s Sexiest Man Alive back in 1987.

“I haven’t changed at all, it’s amazing what happens when you become the Sexiest Man Alive – you get to be stuck in time forever,” he told IMDb Asks. “I just worked with Nick Nolte a couple weeks ago on something and he was also part of that little club, and it is a little club. And you know we don’t get together. We should get together and party right? We should get together and say ‘Hey man you’re pretty sexy aren’t you? Me too!’ “