"We've got to understand each other. We've got to have communication," Randy Jackson says of the violence in Baton Rogue
Credit: Rob Latour/REX/Shutterstock

Just weeks after the police shooting of Alton Sterling and days after the subsequent police ambush that left three Baton Rouge officers dead, Randy Jackson, music producer and former American Idol judge, is speaking out about the violence in his hometown.

“I am just horrified with what’s going on in the city,” Jackson, 60, tells PEOPLE at the 10-year celebration of his eyewear line, Randy Jackson Eyewear. “Black lives matter but all lives actually matter. One injustice doesn’t just make another [injustice] go away.”

The Grammy winner says he doesn’t agree with the “eye for an eye” mentality and hopes for equality and peace in Baton Rouge and the rest of the world.

“My heart and soul goes out to all the people there. We’ve got to get it together. We’ve got to understand each other. We’ve got to have communication. Two wrongs don’t make a right,” Jackson says.

Although Jackson walked away from American Idol in 2014, he’s staying busy thanks to his collaboration with Zyloware, which creates fashionable and affordable eyewear for men. He’s also producing new television shows, movies and music for upcoming artists.

Even with all his new projects, the former session guitarist still misses sitting behind the Idol judges’ table – but not for the reason you might think.

“I would say I miss some of the talent discovering, but I’ve done that all of my life. What I miss is the comradery with the team,” Jackson admits. “Me, Simon [Cowell], Paula [Abdul], all the producers and all the people involved – it really developed into family over the years.”

Jackson says the crew still stays in touch and gets together on occasion, but it doesn’t compare to the good old days.

“There’s nothing like that togetherness that we had on the road doing those auditions, having fun and having some laughs.”