Even in his last years, Jerry Lewis had the biting humor he built a career on

Even in his last years, Jerry Lewis had the biting humor he built a career on.

Lewis died Sunday at the age of 91 from cardiac disease but his agent of 30 years, Jeff Witjas, tells PEOPLE even though his health declined his humor stayed intact.

“It was tough to see him get older, because at the end he couldn’t walk as well and he was in a wheelchair,” Witjas tells PEOPLE. “I never saw Jerry as fallible or vulnerable too much, he always exuded a certain strength and charisma. At the end, at 91, he lived a great life — but it was hard to see somebody you love sort of start breaking down.”

Credit: Rich Fury/Invision/AP

Lewis in August 2016

But through it all, Lewis was still the funniest person in the room. Witjas recalls the last time the comedian visited his offices and was still cracking jokes and giving everyone a hard time.

“Jerry didn’t change much. He lived the life he wanted to live, which was fabulous,” says Witjas. “The joy that he brought to millions of people for so long was incredible. He was always my idol growing up.”

Apart from his iconic career, Lewis also had a reputation for being difficult and often harsh. But Witjas says he saw a different side of the actor that people don’t get to see — his personality away from the spotlight.

“There were some detractors along the way, but they never really knew the man,” Witjas argues. “He always had a terrific heart and he was always very warm. He had his moments every once in a while, but that wouldn’t define the man. I defined him as a loving, giving, charismatic. Jerry was honestly a gem.”

Credit: Jeffrey Mayer/WireImage

Lewis with his daughter Danielle and wife SanDee in 2014

One of the ways Lewis showed his warmer side was by hosting his Muscular Dystrophy Telethons, which raised $2.6 billion over the years. Witjas remembers the comedian taking the telethons seriously and working on them all year, often taking the time to visit kids and adults at hospitals around the country.

“He did it genuinely, there really wasn’t any other reason other than trying to help these kids,” Witjas recalls. “He put a lot of work into it, this was his child. There was always a softness to Jerry that a lot of people didn’t see. There was a vulnerability.”