Comedian Ricky Gervais recounts living in a cramped London apartment with college sweetheart Jane Fallon to PEOPLE's Editor-in-Chief Jess Cagle

From enduring a desk job that eventually inspired The Office to hosting the Golden Globes four times, Ricky Gervais has had his girlfriend Jane Fallon at his side throughout his entire journey.

The couple of 35 years even survived living in cramped quarters in London following college graduation.

F:PHOTOMediaFactory ActionsRequests DropBox48327#Getty imagesGettyImages-599738690.jpg
Credit: Karwai Tang/Getty

“Our first flat was a one room in a place called Kings Cross, which was a really rough area,” Gervais, 56, tells PEOPLE’s Editor-in-Chief Jess Cagle in the latest episode of The Jess Cagle Interview (streaming now on People TV). “I think it was above some sort of seedy sauna . . . That’s all we could afford. Our little bed was in this room. I could open the fridge from the bed.”

The apartment’s other amenities were even less desirable. “It was a shared toilet with the other flats so if I needed a wee in the night, I just popped in the sink,” the comedian, now on his Humanity tour, recalls. “It was closer. I remember once Jane, in her hazy state, just going, ‘Oh at least take the dishes out first.'”

After a failing at career as a pop star and working his way up at a corporate job, the Emmy winner began to get recognized in the U.K. as the host of a comedic talk show. But it was his series The Office — which Gervais wrote, directed and starred in — that finally got the British actor recognized by Hollywood at age 40.

RELATED VIDEO: Ricky Gervais Says He Wants to Host the Golden Globes Again – and Reveals Which Celebs Got Mad about His Zingers

Fallon, who’s an author and now 56, knew why it took Gervais as long as it did to find his niche.

“I said to Jane, ‘Why didn’t I do this earlier?'” he says, referring to the workplace sitcom. “And she said, ‘Cause you wouldn’t have been any good at it.’ And I think it did take till I was 40 to have a voice, and to know how to deal with all this, and to do it for the right reasons. I think if I’d have done it at 21, it’d have been over in a year maybe.”

Asked for his takeaway from his unconventional rise to fame, Gervais mused, “Kids, wee in the sink, do nothing until you’re 40 and then become an international comedian.”

Gervais can currently be heard on his new SiriusXM show “Ricky Gervais Is Deadly Sirius,” a one-hour program that airs weekly on Comedy Greats channel 94.