Jerrod Carmichael Shares How Coming Out as Gay Impacted His Bond with His Mom: It's 'Tricky'

"She didn't account for a gay son. She doesn't really know what to do with it," Jerrod Carmichael said

Jerrod Carmichael's familial dynamics were impacted in differing ways after opening up about his sexuality.

During an appearance on the Howard Stern Show, the Carmichael Show alum said that coming out as gay caused friction between himself and his mother.

"Me and my mom have a God-sized wall between us. You know, that's the impossible thing to navigate," he said, citing his mother's deep ties to Christianity.

But for Carmichael, Christianity became a source of "self-hate" and ultimately kept him in the closet longer. "That's what was really destructive and it's been very, very difficult," he said.

"I'm trying to trust fall into a new interpretation of religion because I do think it has value," he continued. "Like, I haven't completely let go of it and I'm trying to reinterpret it for me and I've been trying to get my mother to have some type of openness to her interpretation of it and it's hard. It's a tricky thing because people — my mother, not just people. My mother holds onto religion because it got her through really rough times."

Since Carmichael's mother grew up in the projects, he recognizes that she "needed God for every step of her life."

"She didn't account for a gay son. She doesn't really know what to do with it," he said. "I can feel her being lost. I want to work with her but talking to her and not acknowledging that part of myself causes a reverberating effect of self-hate that I can't afford right now. So, I added distance. I think it's important."

Danny Matson/Getty

Carmichael used to speak to his mother daily before coming out. However, they no longer maintain constant communication since the comedian doesn't "want the conversation to be fake."

"I'm also trying to eliminate that from my life and eliminate it from my work and just be truthful," he continued, adding that discussions of his sexuality are currently "off the table" with his mother.

Though Carmichael's mother is having a harder time accepting his sexuality, his father has been supportive of him. The actor's dad has even served as a "referee" between himself and his mother.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Carmichael recounted a recent conversation that he had with his father about his mom. "'Your mom called the other day, so take some time to call her. She loves you and misses her baby boy,'" he recalled. "What I wrote back was, 'I know. I love her too, but her baby boy is now a big gay man and I think we need some time to come to terms with that.' And that's where we are."

Though Carmichael has previously said he had been physically intimate with men, he didn't come out publicly until his HBO special, Rothaniel, was released earlier this month. The stand-up performance was taped in front of a live audience in February.

In Rothaniel, Carmichael said he had "a secret" that he kept from his family, friends and fans. "And the secret is that I'm gay," he said.

Timothy Hiatt/Getty

"I'm accepting the love, I really appreciate the love. My ego wants to rebel against it," he said as the audience applauded him. "I rebelled against it my whole life. I never thought I'd come out. I didn't think I'd ever, ever, ever come out. Probably at many points I thought I'd rather die than confront the truth of that, to actually say it to people. Because I know it changes people's — some people — it changes their perception of me. I can't control that."

Carmichael next addressed his sexuality during his Saturday Night Live hosting debut, where he joked about the perks of being gay while living in New York City.

"If you say you're gay in New York, you can ride the bus for free and they just give you free pizza," he said in his monologue. "Honestly, if you're gay in New York, you get to host Saturday Night Live. This is the gayest thing you can possibly do. Like, I came out right onto the stage. We're basically in an Andy Warhol fever dream right now."