The passing of Gyllenhaal's Brokeback costar is a "major trauma," says a source

Heath Ledger‘s sudden death has been especially tough on his good friend – and Brokeback Mountain costar – Jake Gyllenhaal.

The actor, who is godfather to Ledger’s two-year-old daughter Matilda, has been devastated by last week’s news. Says one Gyllenhaal friend, “Jake is taking this harder than most people.”

Nowhere is that more apparent than on the New Mexico set of Gyllenhaal’s latest movie, Brothers.

“This has had a strong personal effect on [Jake],” says a set source. The insider adds that Gyllenhaal left the set immediately after learning of Ledger’s Jan. 22 death – but he flew back on a commercial flight to shoot an additional scene on Thursday.

“He was there, but he wasn’t with us. It was obviously a major trauma,” says the movie source. “These guys were very close. [Jake] was sitting in the director’s chair staring off into space.”

Gyllenhaal has been off since shooting that scene, and was photographed Saturday looking subdued with girlfriend Reese Witherspoon in Los Angeles. Set sources say director Jim Sheridan is shooting around the grieving actor during his personal leave.

The 27-year-old’s somber mood is a sharp contrast to the happier times on the set of Brothers. During one intense early prison scene, Gyllenhaal jokingly reached into his pocket and took out a picture of his Brokeback beau to stick on the prison wall. “Like those prisoners put [loved ones] on the wall, but Jake’s was Heath Ledger,” one set source recalls. “That was hilarious. It was a nice moment.”

The source adds, “When you think back on it now, it’s touching.”

• Additional reporting by ALYSSA SHELASKY