Johnny Depp Says His Mom Was 'Violent' During His Childhood: 'Tried to Stay Out of the Line of Fire'

"The verbal abuse, the psychological abuse, was almost worse than the beatings," Depp said in his testimony about his late mother

Johnny Depp talked about his turbulent childhood while testifying on Tuesday at his defamation trial against ex-wife Amber Heard.

The actor, 58, said his late mother Betty Sue Palmer was "violent" and "cruel" to him when growing up, to the point where he "tried to stay out of the line of fire" where she was concerned.

"The verbal abuse, the psychological abuse, was almost worse than the beatings. The beatings were just physical pain. The physical pain, you learn to deal with. You learn to accept it. You learn to deal with it," said Depp.

Some of the violence Palmer inflicted, Depp alleged, included throwing an "ashtray," "a high-heeled shoe" or "a telephone." He said she would hit her children in their heads, and "had the ability to be as cruel as anyone can be."

For more on Johnny Depp's defamation trial against Amber Heard, listen below to our daily podcast PEOPLE Every Day.

Depp recounted that his father, John Depp Sr., "is a very kind," "quiet" and "shy" man, and "not a confrontational person in any way."

The actor continued, "And when Betty Sue, my mother, would go off on a tangent toward my father — and of course, in front of the kids, it [didn't] matter to her — he, amazingly, remained very stoic and never, as she was rationing him with horrible things, he stood there and just looked at her while she delivered the pain, and he swallowed it. He took it."

Depp went on to say that "there was never a moment when my father lost control and attacked my mother or hit my mother or even said a bad thing to my mother."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human-interest stories.

Johnny Depp with his mother Betty Sue Palmer. Jim Smeal/Ron Galella Collection via Getty

"The things that I witnessed … there were a couple of times when it got too far when I could see his eyes welling up as he was staring at her, saying nothing," the Pirates of the Caribbean star said. "The most he would do is he would punch a wall. I once saw him punch a wall and he shattered his hand because it wasn't drywall, it was proper concrete … but still, never touched her, never argued with her. He remained a gentleman."

"To me, as a 5-year-old boy, I kept wondering, why does he take it? How does he take this? And why doesn't he leave her? But he didn't," Depp added. "He was able to maintain his calm, and his composure. He was able to maintain his relationship with his children. ... He is a good man."

Johnny Depp. JIM WATSON/POOL/AFP via Getty

Depp's comments come a week after his sister Christi Dembrowski testified in court about being abused by Palmer while growing up. While on the stand April 12, Dembrowski said she and Depp "would run and hide" upon seeing their mom being abusive toward their dad, whom she said wouldn't react.

Added Dembrowski, 61, "She would hit us. She would throw things." When asked if Depp ever hit back at his mom when she was abusive, Dembrowski said no, "He never went to that place."

"Really early on as a young child," she said, "none of what was happening in our home felt good. And so, as I got older, both Johnny and I actually, we decided that once we left, once we had our own home, we were never going to repeat, ever, anything similar in any way to our childhood. We were gonna do it different."

Palmer "softened" as she grew older, Dembrowski said. She died at age 81 in May 2016, just days before Heard, 35, filed for divorce from Depp.

RELATED VIDEO: Amber Heard Accuses Johnny Depp of "Sexual Violence" at Defamation Trial, He Calls Claim "Fictitious"

Depp is suing Heard for defamation over a 2018 op-ed she wrote for the Washington Post about surviving domestic violence, though she never mentioned Depp by name in the article. The actor — who also accused Heard of throwing a vodka bottle at him back in 2015, claiming it severed his finger — originally filed the $50 million lawsuit in March 2019 and has argued that being painted as an abuser has tarnished his Hollywood career.

The exes have appeared in person for the hearings since last week, listening to the proceedings from either sides of the courtroom. The trial — which has already seen testimonies from Depp's witnesses like their former marriage counselor, his personal doctor and more — is being televised live via various outlets, including Court TV.

During opening statements last week, Heard's attorney Ben Rottenborn said evidence will show she suffered domestic abuse by Depp that "took many forms," including physical, emotional, verbal and psychological, as well as "sexual violence at the hands of Depp." (A spokesperson for Depp denies the allegation as "fictitious.")

If you or someone you know has been a victim of sexual abuse, text "STRENGTH" to the Crisis Text Line at 741-741 to be connected to a certified crisis counselor.

If you are experiencing domestic violence, call the National Domestic Violence Hotline at 1-800-799-7233, or go to thehotline.org. All calls are toll-free and confidential. The hotline is available 24/7 in more than 170 languages.