"He has never been able to visit their graves," Polanski's lawyer Harland Braun tells PEOPLE 

Nearly 50 years after Charles Manson’s followers brutally murdered nine people, including pregnant actress Sharon Tate, her husband Roman Polanski still hasn’t made a visit his late wife and child’s graves.

After Manson’s death Sunday at the age of 83, Polanski’s lawyer Harland Braun tells PEOPLE the director still longs to visit his wife and child’s burial plots in Los Angeles. The Oscar winner has not returned to the United States since fleeing the country in 1978 after he pleaded guilty to raping a 13-year-old.

Roman Polanski
| Credit: Roman Polanski attends a press conference after the announcement at the regional court in Krakow on October 30, 2015 not to extradite him to the United States to face sentencing for raping a 13-year-old girl in 1977. AFP PHOTO/JANEK SKARZYNSKI (Photo credit should read JANEK SKARZYNSKI/AFP/Getty Images)

“It is a very painful thing because Sharon and his son are buried here in Los Angeles at Holy Cross Cemetery and because of this case, he has never been able to visit their graves,” says the attorney.

A career criminal who spent over half his life in prison, Manson became the leader of a group of young followers he convinced to murder for him and who became known as the Manson “family.”

The savage slayings — committed at Manson’s behest and for which he was found legally responsible but were not, technically, committed by him directly — shocked and terrified the nation.

Manson was eventually sentenced to death in 1971 for conspiracy to commit seven murders, but the sentence was reduced to life in prison one year later when California abolished the death penalty.

Sharon Tate and Roman Polanski
Sharon Tate and Roman Polanski (1966)
| Credit: Hulton-Deutsch Collection/Corbis/Getty

After the killings, Braun, who was a deputy district attorney in L.A. before becoming a criminal defense lawyer, was tasked with prosecuting some of the Manson family after a few of his followers, who had not been charged for the murders, went on a robbing spree in an effort to earn money to free Manson from prison.

“No matter what you would ask, he would say, ‘Thermometer,'” Braun said of cross-examining Manson. “It was bizarre because who would ever put Charles Manson on as a character witness? The whole thing was pretty bizarre.”

While the murder spree captivated the country and made Manson’s name synonymous with evil, Braun says he tries to keep the crimes in perspective. “He is evil but there is a lot of evil in the world,” he says. “You see people being slaughtered by ISIS, children dying in the Mediterranean and he is just part of human nature. There are a lot of evil people around.”

Charles Manson in Court
Charles Manson
| Credit: Bettmann/Getty

He adds, “The sad thing was most of the people in the Manson family had lost their minds in this cultish kind of behavior. It is sad there are evil people out there like that.”

Meanwhile, Polanski recently spoke out about his 1970s rape accusation in a rare interview, saying that he feels he’s completed his sentence.

“As far as what I did: It’s over. I pleaded guilty,” Polanski told The Hollywood Reporter. “I went to jail. I came back to the United States to do it, people forget about that, or don’t even know. I then was locked up here [in Zurich] after this festival. So in the sum, I did about four or five times than what was promised to me.”

Grab Cut Insert Cut Grab Cut Insert Cut F:PHOTOMediaFactory ActionsRequests DropBox29688#apAP690809065.jpg
Body of actress Sharon Tate being taken from her rented house on Cielo Drive in 1969
| Credit: AP

In 2009, Polanski was detained by Zurich police after the United States attempted to have him extradited to face charges relating to the decades-old rape case.

After fleeing the United States in 1978, he was detained in Switzerland, jailed and placed on house arrest for several months, until the Swiss government officially declined to deport him. Since then, Polanski has remained a fugitive. The case is still ongoing, despite his victim’s requests to have it dismissed.

Just a day after Polanski declared he had paid his debt for the rape charge, The New York Times reported that a German actress filed a report with the Swiss police accusing Polanski of raping her at a house in Gstaad when she was 15 in Feb. 1972. According to the outlet, the public prosecutor’s office is still deciding if she can pursue a criminal complaint. A lawyer for Polanski reportedly declined to comment at the time.