"The thing that bothered me at the time was just the commitment of endless movies of that same character over and over," the actor explained
Viggo Mortensen; Hugh Jackman X-Men
Credit: Sylvain Lefevre/WireImage; Attila Dory/20th Century Fox/Marvel Ent Group/Kobal/Shutterstock

Viggo Mortensen is opening up about his decision to turn down a role in the X-Men franchise nearly two decades ago.

The actor, 62, appeared on last week's episode of the Happy Sad Confused podcast where he revealed that he was initially offered the role of Wolverine in X-Men but didn't take the gig over a fear of a "commitment of endless movies."

Mortensen, who despite his commitment worries went on to star in the Lord of the Rings franchise for several years, said he was "bothered" by the idea of playing the "same character over and over."

"I was nervous about that," he told host Josh Horowitz.

The actor explained that his then-preteen son Henry also had a major influence on him saying no to the X-Men films.

Mortensen said that at the time, Henry was a big fan of the Marvel comic books and the star wanted his input on the film.

"There were some things, I mean they straightened most of them out, but I did take Henry to the meeting I had with the director...as my sort of good luck charm and guide," Mortensen said.

Want to get the biggest stories from PEOPLE every weekday? Subscribe to our new podcast, PEOPLE Every Day, to get the essential celebrity, entertainment and human interest news stories Monday through Friday.

The actor recalled letting Henry read the script and his son saying, "This is wrong. That's not how it is."

"He goes, 'He doesn't look like this.' And all of a sudden the director is falling all over himself and then the rest of the meeting was him explaining in detail to Henry why he was taking certain liberties," he said. "We walked out of there, and [Henry] asks if they will change the things he told him about, and I say, 'I don't think so... I'm not going to do it anyway, because I'm not sure I want to be doing this for years.'"

"And then a couple of years later I'm doing three Lord of the Rings, so who knows," Mortensen said with a laugh.

While Mortensen went on to play Aragorn in the Lord of the Rings franchise, Hugh Jackman landed the role of Wolverine.

"I think he did great, I'm sure no one can imagine anyone doing it better than he did," Mortensen said of Jackman's portrayal.

Mortensen's directorial debut Falling is now in theaters, on digital and On Demand.