Seth Rogen adds that his wife Lauren Miller, whom he wed in 2011, "wants kids less than I do"

Seth Rogen is getting honest about not having a desire to become a father.

Appearing on The Howard Stern Show this week, the Knocked Up actor, 39, explained why he and wife Lauren Miller both don't plan on having kids. Rogen says he wouldn't be able to work as much as he would like if he also had to juggle parenting responsibilities.

"I wouldn't be able to do all this work that I like," he says. "... People are always like — it's something I think I was uncomfortable answering this before — but they were like, 'How do you do so much?' The answer is I don't have kids. ... I have nothing else to do."

As for Lauren, whom he wed in 2011: "I would say she wants kids less than I do. I could probably be talked into it; she's like no."

The Yearbook author adds that he and his wife "have so much fun."

"I don't know anyone who gets as much happiness out of their kids as we get out of our non-kids," says Rogen with a laugh. "Like, we're f---ing psyched all the time! We're laying in bed on Saturday mornings smoking weed, watching movies naked. If we had kids, we could not be f---ing doing this."

Want to get the biggest stories from PEOPLE every weekday? Subscribe to our new podcast, PEOPLE Every Day, to get the essential celebrity, entertainment and human interest news stories Monday through Friday.

Seth Rogen, Lauren Miller Rogen
Seth Rogen and Lauren Miller Rogen
| Credit: Roy Rochlin/Getty

Stern, who has three kids, tells Rogen, "It's not an easy haul having kids."

"You can't be narcissistic, you have to give yourself over to parenting, and they've gotta become the priority," says Stern, 67, to which Rogen replies, "Yeah, like, I don't want that. That does not sound fun to me."

"And why?" the Neighbors actor adds, "There's enough kids out there. We need more people? Who looks at the planet right now and thinks 'You know what we need right now? More f---ing people.' That's truly a confounding thing to me."

Celebrating nine years of marriage and 15 years together in October, Rogen wrote on Instagram that he "can't believe I've been lucky enough to find the perfect partner to go through life with." He added that he loves her "more every day."

Rogen said on the Armchair Expert podcast in 2018 that potentially having kids was an "active conversation" with Lauren, adding that he thought he "would be good at" being a dad.

"We've gone from people who just haven't had kids to people who don't have kids," he said at the time. "It honestly makes it more fun in some ways because it's like, 'Man, no one we know is doing this s---. We can just do whatever we want!' Not that we even do that much."

"A lot of people [try to convince us to have kids] because they seem to like their kids," Rogen added. "People get joy from it and they feel as though I don't have that joy. I appreciate that. I'm pretty at peace."