As sexual assault trial looms, wife of beleaguered accused comedian stands by her husband of 51 years
Credit: Evan Vucci/AP

When a Pennsylvania District Judge ruled on May 24 that entertainer Bill Cosby will stand trial on sexual assault criminal charges, it was a huge legal blow to the 78-year-old comedian. But the day-to-day fall-out from the case had long since taken a toll, a source close to the Cosby family tells PEOPLE.

“He doesn’t have his life anymore,” the source says.

“He has not lost his sense of humor and he has not become morose. But he is somber and there’s been a physical toll – his walking is halting and he can seem downtrodden.”

While Camille Cosby, 71, continues to stand by her husband of 51 years even as more than 50 women have accused him of sexually assaulting and/or drugging them over the years, the ongoing ordeal has not been easy. “Of course it’s very painful for Camille to go through this,” says the source.

“They have been married for over 50 years and she loves him For both of them, it’s a nightmare every day.”

For more on Andrea Constand, the woman at the center of the criminal case against Bill Cosby, pick up PEOPLE on stands Friday

The nightmare won’t be over any time soon.

With Cosby facing criminal charges and a trial for allegedly drugging and sexually assaulting Andrea Constand, 43, while she was a Temple University employee in 2004, the case will certainly drag on for the foreseeable future.

Cosby and his attorneys have steadfastly denied the allegations from Constand, insisting the sexual contact was consensual, as well as those leveled by dozens more accusers. (While Cosby is fighting other civil cases involving similar accusations, he has only been charged criminally in Constand’s case.)

New revelations came to light on May 20 when a federal judge unsealed Cosby’s depositions in Constand’s civil suit, which he settled with her in late 2006. The depositions, which now are apart of a defamation lawsuit filed against Cosby by seven women claiming he drugged and sexually assaulted them, include his admission that he gave quaaludes to women and had sex with at least two teenaged girls.

As long as the legal case continues to play out, Cosby is expected to remain largely out of the public eye – and, to his disappointment, off the stage. “One of his greatest joys is performing and he can’t do that or play an arena anymore,” the source tells PEOPLE.

“That’s been taken away from him.”