Adam Driver Jokingly Recalls 'Scary' Comic-Con Experience: 'I'm Not Anxious to Go Again'

"It's just a lot of energy, as you can imagine," Star Wars alum Adam Driver said of the massive fan event

Adam Driver isn't itching to make an appearance at Comic-Con again.

The 37-year-old actor and his House of Gucci costar Lady Gaga appeared on the U.K.'s Graham Norton Show Friday. During the episode, Driver reflected on the time he attended Comic-Con while promoting his Star Wars films. When host Norton directly asked whether Driver liked being at the fan convention, he answered simply, "No."

"I didn't know I was not gonna like it until— I didn't know the rules of a Comic-Con," he explained. "I got in at the hotel at 2 in the morning ... [and I'm like], 'Maybe tomorrow I'll go get a coffee,' and they're like, 'Oh no, you can't get a coffee.' I'm like, 'Well, maybe I'll get a coffee in the hotel.' They're like, 'No, you can't get a coffee in the hotel.' "

Driver said he was told in order to leave his hotel room, he should wear a mask to hide his identity — and they offered him a Darth Vader or Iron Man mask.

"If you want to go outside, they're like, 'Just put a mask on so nobody knows who you are,' because Star Wars [The Force Awakens] hadn't come out yet," he said. "Then I opened my window 'cause I'd been in the room for 24 hours before this thing we were supposed to do, and there was the band at the bottom of the building playing the Star Wars theme on repeat because we were all staying in the hotel. It was scary."

"It's just a lot of energy, as you can imagine," Driver said of the thousands of "devoted" fans who flock to the event.

The star said he's been to Comic-Con "only the once."

"I saw what it was. I mean, it's nice. I'm not anxious to go again."

Adam Driver at San Diego Comic-Con in July 2015. Jesse Grant/Getty

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

After taking the role of Kylo Ren in the Star Wars franchise, Driver skyrocketed to a new level of stardom. In 2017, he talked to The Guardian about how his life had changed.

"As a person, I'm the same. The problems I had before Force Awakens, it didn't solve any of them," he said at the time. "For me, the only noticeable difference is your visibility as a person. Loss of anonymity is a big thing. I didn't realize how I would see that in a billion little ways."

He continued, "[...] part of my job is being anonymous and I think being able to live, to observe more than to be observed, is important. [Being famous] seems counterintuitive to my job. It's a weird dynamic when you walk into a room and there's an image people project onto you."

"What it means to lose anonymity is a bougie problem in and of itself," Driver said. "And I won't garner sympathy, nor am I asking for it. The image of us on our red carpet wearing expensive suits, where people naturally assume your life is, is not what I was after when I started this job. Believe it or not."

House of Gucci is in theaters Nov. 24.