"I would be saddened that I would have to say to no to someone who did so much for me all those years ago," Daniels said of working again with Allen

Jeff Daniels remains conflicted about Woody Allen.

“I named my theater after a movie I did with Woody Allen called the Purple Rose of Cairo and I’m not changing the name,” Daniels told Sunday Today host Willie Geist in an episode airing Sunday, March 11 at 8am ET/PT on NBC.

“That movie meant the world to me,” he explained. “It was a turning point in my career. I was working with one of the great American filmmakers.”

However, the actor noted that Allen, 82, “is now one of the great American filmmakers with a big asterisk next to that.”

Jeff Daniels (L) and Woody Allen
| Credit: en Gabbe/Getty; Marc Piasecki/WireImage

As for whether he would work again with the director, Daniels said, “If Woody were to call me and say, ‘I want you to do a role in my next movie,’ the easy answer is to say no. The harder answer is, ‘I don’t know.’ Because I would be saddened that I would have to say to no to someone who did so much for me all those years ago.”

Daniels, 63, previously said that he does believe the accusations of sexual assault made against Allen by his daughter, Dylan Farrow. In 2014, Farrow – who is one of the star’s three children with ex Mia Farrow — publicly claimed that Allen molested her as a child. Allen has long denied the allegations, which first surfaced during his explosive 1992 split from Mia. The director was not charged, though a Connecticut prosecutor said there was probable cause for a criminal case.

RELATED: Woody Allen Responds to Dylan Farrow Interview Where She Claims ‘He’s Been Lying for So Long’

Allen’s son Ronan Farrow also wrote a column for the Hollywood Reporter, which addressed the sexual abuse claims and condemned the media for not asking Allen about the allegations and stars for working with him. In response, Allen told The Guardian, “I have no interest in all of that. I find that all tabloid stupidity.”

The controversy surrounding Allen has made headlines again after the Harvey Weinstein sexual harassment accusations rocked Hollywood. A number of celebrities have recently come out and expressed regret for working with the Annie Hall director, including Colin Firth, Rachel Brosnahan, Timothée Chalamet, Mira Sorvino and many more.

Weinstein, meanwhile, has admitted to making advances on actresses, but vehemently denies allegations of sexual harassment and assault. A spokesperson for the producer previously told PEOPLE in a statement that “any allegations of non-consensual sex are unequivocally denied by Mr. Weinstein. Mr. Weinstein has further confirmed that there were never any acts of retaliation against any women for refusing his advances.”