January 24, 2018 03:11 PM

James Franco continues to struggle after several women accused him of sexually inappropriate behavior.

Sources tell PEOPLE that the actor, who was just snubbed by the Academy Awards for his role in The Disaster Artist, isn’t doing well in the aftermath of the allegations.

“His team wants him to continue making public appearances. He thinks he can save his image,” one of the sources says. “He looks like he hasn’t slept for days. He’s just a mess.”

Echoes a second source, “This whole process has been very hard on him. He’s been shaken up.”

However on Tuesday, Franco was spotted out grabbing lunch with friend Jeff Garlin in Los Angeles. The actor kept his head down as he left celebrity hotspot Jones on Third just hours after the Oscar nominations were announced.

Franco skipped the Critics’ Choice Awards on Jan. 11, just a few days after the allegations first surfaced, where he won best actor in a comedy. He did, however, attend the Screen Actors Guild Awards the next week with his girlfriend Isabel Pakzad. The actor lost to Darkest Hour‘s Gary Oldman that night after skipping the carpet and going straight into the show.

WATCH: James Franco Skips Critics Choice Awards as Source Says He’s ‘in a Really Bad Place’ After Allegations

Before his SAG Awards appearance, a source told PEOPLE that he was trying to keep a low profile.

“He’s in a really bad place, so bad that he changed his phone number,” a source close to the situation previously told PEOPLE. “His close friends are trying to be there for him but it’s been hard – he’s only talking to a select group of people. For now, he’s just hiding out.”

During an appearance on Late Night with Seth Meyers, the actor addressed the sexual harassment allegations after wearing a Time’s Up pin at the Golden Globe Awards.

“There are people that need to be heard,” he told Meyers. “I have my own side of this story, but I believe in these people that have been underrepresented getting their stories out enough that I will hold back things that I could say, just because I believe in it that much,” he said. “So if I have to take a knock because I’m not going to try and actively refute things, then I will, because I believe in it that much.”

Franco at the SAG Awards Jan. 21
Dimitrios Kambouris/Getty

In an article published in the Los Angeles Times, five women accused the actor of abusing his power as an acting teacher and mentor in a sexually exploitative manner.

Two students claimed the actor would often become angry on set when they would refuse to film topless while another former acting student at the film school Franco founded said he once removed safety guards while filming an oral sex scene on the set of the 2015 film The Long Home.

Additionally, actress Violet Paley recounted her previous social media claims that Franco exposed himself and tried to pressure her into oral sex. Though she said they had a consensual relationship, Paley said: “that time wasn’t consensual.” She also alleged that he told her friend to meet him in a hotel when the friend was 17.

Franco’s attorney, Michael Plonsker, denied each of the women’s allegations, and cited Franco’s comments on The Late Show With Stephen Colbert as his formal denial.

“Look, in my life I pride myself on taking responsibility for things that I have done,” Franco told Colbert. “I have to do that to maintain my well being. The things that I heard that were on Twitter are not accurate. But I completely support people coming out and being able to have a voice because they didn’t have a voice for so long. So I don’t want to shut them down in any way.”

You May Like