For host Jimmy Kimmel, the Oscars envelope mixup didn't just cause of winners from La La Land and Moonlight to be confused, it also caused his final bit to be cut

The ending of Sunday’s Academy Awards became one of the most memorable moments in Oscar history — when Bonnie and Clyde costars Warren Beatty and Faye Dunaway opened the wrong envelope and mistakenly announced La La Land as the Best Picture winner instead of Moonlight, the actual victor.

The fiasco was cleared up onstage at the Dolby Theatre a few minutes later, and accounting firm PricewaterhouseCoopers accepted responsibility — blaming its envelope guardians, Brian Cullinan and Martha Ruiz.

But for host Jimmy Kimmel, the mixup caused his final bit to be cut.

The late-night host, 49, gave an interview to The Bill Simmons Podcast on Wednesday and he described one last dig he was planning on making at Matt Damon‘s expense.

The moment would have been a revenge of sorts for the Emmys, where Damon teased Kimmel about losing his category.

Credit: Chris Pizzello/Invision/AP

“The plan for the last joke was — the Best Picture winner is announced, they make a speech [and] I was assuming the Best Picture was not going to be Manchester by the Sea,” Kimmel explained. “I said if it’s not Manchester by the Sea, I’m going to be sitting next to [Damon], and it’s going to be my revenge for the Emmys.”

He continued: “I’d start wrapping the show up, but then the camera would widen, and we’d see that Matt would be sitting next to me and I would say, ‘Well, you know, it’s unbelievable, Casey [Affleck] won and Kenneth [Lonergan] won. There’s only one person who didn’t win tonight and it’s you.’ “

“It would have been a nice little button to the whole night,” Kimmel joked.

Credit: Chris Pizzello/Invision/AP; David Fisher/REX/Shutterstock

Kimmel was next to Damon when the final bit went down, as he previously said during his Monday night Jimmy Kimmel Live! monologue.

“But we’re sitting there, and we notice some commotion going on, and Matt says, ‘I think I heard the stage manager say they got the winner wrong,’ which is unusual,” Kimmel said. “You figure, ‘Well the host will go onstage and clear this up.’ And then I remember — oh I’m the host!”

RELATED VIDEO: Oscars 2017 Full Fashion Recap

Once he was onstage, Kimmel joined producers and executives trying to figure out exactly what went down. Kimmel told The Bill Simmons Podcast that he was impressed with Warren Beatty’s presence of mind to hang onto the evidence.

Trying to prove he was not responsible for the error, the 79-year-old wouldn’t hand over the two envelopes.

Check out PEOPLE’s full 2017 Academy Awards coverage and complete winners list!

“He did a very smart thing, and I was impressed by this. He kept those envelopes,” Kimmel said. “He would not hand them to anybody. He showed them to people, but he would not hand them over. The two envelopes — he kept both of them. Because that’s the evidence, that’s the smoking gun there. As soon as you give those envelopes up, who knows who switches what out, and who knows what happened.”

Sadly, Kimmel said, Beatty declined his offer to appear on his late-night show and explain what was going through his mind.