The actor's Showtime series will return later in the year for it's third season
Credit: Ethan Miller/Getty

When it comes to television, the one show Joshua Jackson doesn’t watch with longtime girlfriendDiane Kruger is his own!

Jackson, 37, stars in Showtime’s hit drama series The Affair, and while the script is steamy, that’s not the main reason he doesn’t tune in with Kruger, 39.

“It’s not the sex scenes,” Jackson told PEOPLE at Monday’s FYC screening and conversation of his drama series in New York City.

“I don’t know why any couple would watch the show together because at some point in every episode there is going to be something that has come up in a fight between the two of you,” he joked. “And you’re going to get the side eye and then it’s like disaster.”

His solution? “Watch separately, talk about it together.”

While the show is currently on hiatus – The Affair returns later in the year for season 3 – Jackson said his fans aren’t afraid to get personal with him about their thoughts on the series’ main topic: cheating.

“I get downloaded on a lot of intimate details of people’s lives,” he said.