Ben Affleck Addresses Jennifer Garner Divorce Comments: 'The Exact Opposite of Who I Am'

"I would never want my kids to think I would ever say a bad word about their mom," said Ben Affleck, who shares daughters Violet, 16, and Seraphina, 12, plus son Samuel, 9, with Jennifer Garner

Ben Affleck is clarifying his recent comments about alcoholism and his marriage with ex-wife Jennifer Garner.

During an appearance on Jimmy Kimmel Live! Wednesday night, the 49-year-old The Tender Bar actor addressed remarks he made on The Howard Stern Show, which included that he'd "probably still be drinking" if he stayed married to Garner, 49.

"I would never want my kids to think I would ever say a bad word about their mom," he told host Jimmy Kimmel. Affleck shares daughters Violet, 16, and Seraphina, 12, plus son Samuel, 9, with Garner.

The dad of three said that he thought his interview with Stern had been "meaningful" and didn't realize how his comments had been perceived until after it aired, adding that some listeners had "taken the conversation" and "made it seem as if I was doing the exact opposite of what I said" by focusing on only one piece of their chat.

For more on Ben Affleck and other top stories, listen below to our daily podcast on PEOPLE Every Day.

In the interview with Stern, he explained, "I had gone on and said how much we respect each other and cared about each other and cared about our kids and put them first." Affleck added that the part of the interview that gained attention "just made me out to be the worst, most insensitive, stupid awful guy."

While he said he understands that his personal life can make headlines or his pictures can be turned into "memes," Affleck explained that when it comes to his family, "I have to draw a line, and be clear."

He concluded, "That's not true. I don't believe that. It's the exact opposite of who I am, what I believe."

Ben Affleck, Jennifer Garner. Magnus Sundholm/Shutterstock; Jon Kopaloff/Getty Images

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

During his Stern interview, Affleck got candid about his struggles with drinking while married to Garner. Recalling how his drinking habits worsened during difficult times in their relationship, Affleck said if they'd stayed together, they "would've ended up … probably at each other's throats."

"I'd probably still be drinking. It's part of why I started drinking … because I was trapped," he admitted. "I was like 'I can't leave 'cause of my kids, but I'm not happy. What do I do?' What I did was drink a bottle of scotch and fall asleep on the couch, which turned out not to be the solution."

The actor/director has gone to rehab multiple times for alcohol addiction over the years, including in 2001, 2017 — which he revealed in a post on Facebook — and 2018.

Affleck, who wed Garner in 2005 then separated in 2015 before finalizing their divorce in 2018, also told Stern that he and the 13 Going on 30 actress did everything they could to keep their marriage together, but it just didn't work out.

"The truth was, we took our time. We made the decision … We grew apart," Affleck explained.

"We had a marriage that didn't work, this happens, with somebody that I love and respect, but to whom I shouldn't be married any longer. Ultimately, we tried, we tried, we tried, because we had kids. Both of us felt like we don't want this to be the model that our kids see of marriage."

While Affleck and Garner had their fair share of ups and downs over the course of their relationship, the former spouses are on good terms now.

Garner even joined Affleck and Jennifer Lopez, who rekindled their romance earlier this year after calling off their engagement in 2004, for some trick-or-treating fun with their kids on Halloween.

RELATED VIDEO: Jennifer Garner Joins Ex Ben Affleck and Jennifer Lopez While Trick-or-Treating with Kids in Malibu

"Everyone gets along and the focus is always on the kids. Jennifer, Ben and Jen Garner all wanted the kids to have an amazing Halloween," a source previously told PEOPLE. "The kids are friends and wanted to go trick-or-treating together. It made sense that they all went together."

Earlier this month, Affleck told WSJ. Magazine about growing from past mistakes.

"There's a lot that I would want my younger self to understand. Some things, honestly, that I'm too self-conscious of or shy about to be really candid about with the whole world because they're mostly mistakes," he said. "Things I wish I had done differently, and they're rooted in that instinct to look at my past and think, I wish I could have avoided this painful event.' I wish I could have not caused someone else pain. I wish I had understood better the nature of what was difficult about life for me."

"I wish I did not have to learn some lessons the hard way," the Gone Girl actor continued. "But maybe this is just rationalizing because the alternative is too painful."

Affleck said he learned one cannot get to true enlightenment "the easy way" and that he "had to learn those things in an authentic, meaningful way to really learn the lessons that I've really internalized, that have created the values that I have now even though most of them were born of failure."

"The only real cure for alcoholism is suffering. You just hope that your threshold for suffering is met somewhere before it destroys your life," said the Oscar winner.

If you or someone you know is struggling with addiction, please contact the SAMHSA substance abuse helpline at 1-800-662-HELP.