"You can't say no to Mariah," Cowell, a huge fan of Carey's, tells PEOPLE

By Lorenzo Benet and Dahvi Shira
Updated August 07, 2011 06:30 AM
Credit: Ferdaus Shamim/WireImage; Jeff Vespa/WireImage

Whatever Mariah Carey wants, Mariah Carey gets.

The X Factor judge Simon Cowell, whose new talent competition debuts this fall, is willing to give into the demands of the pop diva, whom he says “wanted to be involved in the show from day one.”

“You can’t say no to Mariah,” Cowell, 51, tells PEOPLE. “We have to find a role for her. I’m a huge fan of hers – not just as an artist, but as a person, and I love her to death.”

But Cowell also says that having Carey on the show poses a slight problem.

“We just can’t shut Paula [Abdul] and Nicole [Scherzinger] up,” he joked. “Add Mariah into the mix as well and you won’t even have anyone singing.”