Bob Arum tells PEOPLE he believes Muhammad Ali will be remembered more for his actions outside of the ring than his triumphs inside it

By Chris Harris
June 10, 2016 12:00 PM

Three-time world heavyweight boxing champion Muhammad Ali preached nonviolence outside the ring while wielding fists of fury inside it. Subscribe now for an inside look at his remarkable life, plus rare family photos, only in PEOPLE.

For half a century, Hall of Fame boxing promoter Bob Arum could call Muhammad Ali a friend. This week, Arum tells PEOPLE he’s mourning the iconic boxer’s passing – a loss, insists the 84-year-old founder and CEO of promotions firm Top Rank, Inc., that is being felt the world over.

“This story of Muhammad Ali resonates with us all, as people,” Arum says. “Obviously, Ali was in bad health the last years of his life, so to that extent he couldn’t make much of a contribution. But people look back and they see what this man has meant, and it reflects in ways on all of us.”

Decades from now, Arum tells PEOPLE he believes Ali will be remembered more for his actions outside of the ring than his triumphs inside it, noting how, during the height of the fighter’s career, Ali took several stoic stances on a number of social and cultural issues.

“He was a kid that came from a border state and a city that was segregated,” Arum says. “He saw there were white and black sides to restrooms and it was clear to him, and, as a young man who had just won an Olympic gold medal for the United States, this very outgoing and exuberant personality, he comes back to America and it is made known to him, clearly, that you’re a second class citizen.”

From left: Muhammad Ali, wife Lonnie and Bob Arum
Ethan Miller/Getty

“So, he embarks on a boxing career, and he had a very, very good mentor in Angelo Dundee, who took him out of the Olympics and built him up by having him fight different levels of fighters. This helped to get his confidence going. It was a long climb up.”

Arum says he can remember Ali’s first meeting with Malcolm X during the early 1960s, and claims talking to the human rights activist had an indelible impact on the boxer.

“Malcolm X brought out in Ali a philosophy that he, himself, wasn’t quite able to articulate,” Arum says. “After meeting, Ali realized there was no reason for him to be a second-class citizen. The idea that white was good and black was bad was something that had been invented, and Ali learned from Malcolm X that it was OK to speak out about these things.”

For much more about Muhammad Ali’s life and legacy, read the new issue of PEOPLE, on newsstands Friday

Muhammad Ali on the cover of PEOPLE

Arum says he believe Ali was crucial to the galvanization of the civil rights movement, which he claims owes as much to Muhammad as it does to Martin Luther King, Jr.

“Ali took principled stands, essentially based on the second-class citizenship of black people in the United States,” Arum says.

“Martin Luther King was articulating that same principal but doing so in a much milder way and a much more acceptable way to white people than Ali was. So Ali became this symbol and spokesman for black people and for Muslims. When he refused to be drafted into the army, he was interviewed and said, ‘I have nothing against the Vietcong. They never used the word n—-r with me. I am not going into the service.'”

Ali’s refusal to fight for America nearly cost him his career, Arum says. A boxing match he’d organized in Chicago was cancelled after Ali spoke out against the Vietnam War.

“The original Mayor Daly got us kicked out of Chicago,” Arum recalls. “For weeks, we couldn’t get a site in the United States for this fight to happen. We had to do it in Canada.”

Had those who’d opposed Ali’s stance been victorious, the Civil Rights Movement would have suffered a substantial setback.

“Had that fight not happened in Toronto in the March of 1966, it would have been a tremendous blow to the movement,” Arum claims. “It would have demonstrated to people that, if you opened your mouth to speak out against something, you ran the risk of the establishment standing up and depriving you of your livelihood or worse. The fact that this fight happened sent a very good message to everybody — one that Ali continued to make even when he was convicted” for draft evasion, resulting in thousands in fines.

For more on Muhammad Ali, tune in to PEOPLE’s List, Saturday, June 11, 8/7c on ABC

“He took that punishment with dignity and he found a way to support himself, which resonated in the U.S. and with people all around the world,” Arum says. “That is when he became this tremendous world figure. By 1970, when he was making his comeback, he was the most popular athlete and most recognizable person in the world.”

It was that universal awareness of Ali’s greatness that added more weight to his position on racial inequality.

“I really believe as much of a contribution as Martin Luther King made to our society, Ali’s contribution, in a lot of ways, was greater,” the boxing promoter says. “He was much more forceful and he demonstrated to people that it was okay to really speak out, and do so in a strong manner.”