"I sometimes think I have a great love affair with humanity but not too good with humans," the actor shared

sean penn
Credit: Matt Winkelmeyer/Getty

Sean Penn knows he's not everyone's cup of tea.

The 59-year-old actor seems to have earned a reputation in Hollywood for being hard to work with and even referenced it during his acceptance speech for the Best Actor for Milk at the 2009 Academy Awards.

While speaking to SiriusXM's Howard Stern on Wednesday, Penn touched on the reputation again and admitted that it stems from clashing with some of the directors he's worked with.

"There've been several times I've worked with directors who I felt might have found a different job description, and perhaps weren't the storytellers that their initial meetings with each of us actors might have indicated," he said.

"Actors are kind of canaries in the coal mine emotionally, and you have to go to whatever place is necessary inside yourself. If you don't have somebody there who at least respects that — most of what I was referring to is that the arrogance goes further than charm," Penn added.

He continued, "I am aware that I can be a difficult person to like from afar, often. I sometimes think I have a great love affair with humanity but not too good with humans."

Stern then asked Penn — who is dating 27-year-old actress Leila George — if he would consider himself a difficult person to be in a relationship with as well, to which the actor replied, "Yeah, I think so."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories

"I will say I'm a person who feels and expresses energetically, and that has misled people into thinking I think it's all about me," Penn explained. "I think the interview you're talking about would be with a lady named Leila George, she knows me better than I do on this level, that doesn't mean that I agree with her perceptions but I certainly find them fascinating."

"She's the gal in my life," the I Am Sam star said of George, who he’s been dating since 2016.

In April, the Oscar winner was spotted with the actress helping volunteers at a drive-thru coronavirus testing site in Los Angeles run by his nonprofit Community Organized Relief Effort (CORE).

Penn's NGO had teamed up with the mayor’s office and the L.A. Fire Department to run the locations in order to free up healthcare workers needed on the frontlines.

“What is going to be most valuable in this pandemic of what we as citizen organizations can do,” Penn told ABC News at the time, “is to get the professionals relieved of duties that are beneath their training sets … so they can get back on the street.”