Ben Affleck Jokes About Sexiest Man Alive Rivalry with George Clooney: 'He Likes That Stuff'

George Clooney, who directed Ben Affleck in his new film The Tender Bar, was named PEOPLE's Sexiest Man Alive in 1997 and 2006

Ben Affleck and George Clooney are keeping tabs on each other's Sexiest Man Alive covers.

Speaking with PEOPLE (the TV Show!)'s Adrianna Costa at Sunday's Los Angeles premiere of The Tender Bar, Affleck, 49, joked about director Clooney, 60, having one more Sexiest Man Alive title to his name. Affleck earned the designation back in 2002, while Clooney graced PEOPLE's Sexiest Man Alive cover in both 1997 and 2006.

"Well, I passed a few times, obviously," Affleck joked about why Clooney has one more than him. "George, he likes that stuff and he wanted to keep doing it and I felt like maybe one was enough."

He also discussed their competitiveness in both having played Batman on the big screen: Clooney in 1997's Batman & Robin and Affleck in the recent Justice League films. "Yes, two Batmans. Two people who have been in People magazine too — and that's no joke," he said.

RELATED GALLERY: See Every Sexiest Man Alive Cover

L: Caption . PHOTO:

R: Caption . PHOTO:

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Affleck recently told WSJ. Magazine about why he took the role of Uncle Charlie in The Tender Bar, a performance for which he was nominated for best supporting actor at the Golden Globes on Monday.

"Just on the merits of the writing and the quality of the role, anyone's going to want to play this part. Also, George was directing, so it was a no-brainer," he explained. "I thought, I'm lucky to get a crack at this, and it's incumbent not to screw this up. But also, even though the story is set in Long Island, it was oddly close to home."

George Clooney and Ben Affleck. Kevin Mazur/WireImage

"It was oddly close to playing my own father. He was very much like Uncle Charlie," added Affleck. "My father was a bartender. And growing up, I would meet him at the bar. My dad didn't go to college, but he was well-read, and his ambition was to be a writer, although he never found professional success. He was hindered by his alcoholism."

"In some ways we can be our own worst enemies," he continued. "He was a reverse class snob. He held a disdain for wealthy people, which stemmed from his own insecurity about not being wealthy and not being educated formally and a desire to prove himself. Which I found that later in life I subconsciously shared."

The Tender Bar is in select theaters Dec. 17, nationwide Dec. 22, then streaming on Amazon Prime Video Jan. 7.