He has been off American TV since the end of The X Factor USA in 2013.
The music mogul responsible for breaking chart-topping acts including One Direction and Fifth Harmony, tells PEOPLE: “It wasn’t like a major thing, but you get to a point where if you’re too tired and there’s too much pressure, it’s not fun anymore and I just thought I can relieve the pressure for a couple of years, which was a good thing.”
“The interesting thing is everybody thinks X Factor was canceled, that wasn’t the case. What happened was that after three years I said, ‘I can’t do this any more’ – it was just too many shows.”
He adds, “Fox offered us another season, but I couldn’t do it, so that’s why we stopped the production, because I couldn’t do it anymore. Even though the numbers weren’t as good as we hoped, in today’s climate, they were brilliant figures, we just didn’t realize it at the time!”
Continuing to appear on the X Factor UK and Britain s Got Talent as well as produce America s Got Talent and working on other formats under his SYCO Entertainment label, Cowell says: “I started to enjoy doing the U.K. shows again and then I really felt mentally, physically, this is the right time to do this in the States again.”
For more from Cowell, pick up the latest issue of PEOPLE, on newsstands now
Now appearing alongside Heidi Klum, Mel B and Howie Mandel on AGT. Cowell – who says he only took the job after Howard Stern announced he was quitting – admits: “It’s easy to get into the habit where these jobs do become a bit hard work and it shouldn’t be like that.
“If we’re lucky enough to be making these shows they have to be the most fun you’ve ever had and I think we’re getting back to that place where everybody seems to be having a good time.
“I certainly see that on AGT where the people who are on the show are really, really enjoying themselves and that makes a much, much better show when that happens.”