The rapper was handcuffed when police said he matched the description of a gas station robbery suspect

Wyclef Jean is opening up about his recent brush with the police — and is calling for change.

Early Tuesday morning, the rapper was handcuffed by the L.A. County Sheriff’s Dept. when officers mistook him for a nearby gas station robber. But now Jean, 47, is considering a lawsuit.

After spending the day in various recording studios working on new music, Jean tells PEOPLE he arrived at his hotel, where police accosted him.

“The cops pulled up, and they told the driver, ‘Get out.’ When the driver got out, I was in the passenger seat. Guns are drawn, and I wasn’t going to get out of the car until those boys put their guns down. The minute I step outside the car, there’s officers that are already on me, they’re already apprehending me; I see guns on me. They take me — and bring me over to the front of their police car,” says Jean.

The musician says officers did not identify themselves and that when he exited the car and was cuffed, he asked people around him to film the incident on their phones.

“I felt, at this point, the only thing that could protect me is the social media. I was like, ‘Roll the cameras, man, because I don’t know what’s going on; I don’t know who these people are,'” says Jean, who adds he told officers to go to check his ID in his hotel room, where he left it, but that they didn’t.

“They take it a step further, slap the bandana off my head, then they take me and shove me in the back of their police car,” he says. “I’m basically telling them, in the front, ‘This is my name. Google me. Look.’ And there’s an officer outside the car with his hand on his gun, waiting for me to flinch or make some stupid move. So finally another cop car comes and says, ‘Ya’ll got the wrong person.’ I’m taken out of the cop car. And these people just go, ‘Sorry!’ as if nothing happened.”

Jean says he feared for his life during this case of mistaken identity.

“When I got out of the car and they rushed me, I felt like running. You feel instant fear because all you see is guns, and you’re a citizen,” he says. “I thought, ‘Oh, sh-t, this must be my end date.'”

Jean believes that the authorities acted with unjustified force. “I feel like this was a case of police brutality. The idea of you getting cuffed in the back [of the cop car], this is not cool. I feel like this is abuse of power.”

However, he believes the issues run deeper than race. “What I want to do is go past the race card. We — police and black folks — have been getting into it before you and me was probably even born,” he says. “But the problem that I’m having with this is: when the Sheriff’s Dept. [writes in their report], ‘Mr. Jean had his hand by his waist. Mr. Jean was going for his trunk.’ I have a major problem with that,” says Jean, who denies reaching for his waist or the trunk. “Luckily, I’m alive to say, ‘That’s not how it happened.'”

“My whole thing is: I teach my daughter, when police stop you: Be calm, be nice, do not talk back to the officer. So she watched me do all of this stuff on social media and saw her father still in handcuffs. And on top of that, when history is reading, I don’t want her to see, ‘Maybe, Dad, you did resist, or you went to the trunk.’ So I need them to tell the real story of what happened. And this is what makes me upset.”

Remy Martin Presents "The Producers Series: Season 2" Qualifier No. 2
Credit: John Lamparski/WireImage

The Los Angeles County Sheriff’s Department apologized to Jean, saying in a statement:“It is unfortunate that Mr. Jean was detained for six minutes during this investigation, as he had no involvement whatsoever in this violent crime … The Los Angeles County Sheriff’s Department is apologetic for any inconvenience this process caused Mr. Jean. We are grateful we were able to apprehend the robbery suspects and that no one was seriously injured.”

A representative for the agency told PEOPLE the department “stands by our statement” and the language in their report, adding: “During the contact we offered [Jean] our business card to please contact us to discuss the matter further and we have yet to hear from Mr. Jean. We welcome him to still reach out.”

Jean hopes that sharing his story makes a difference.

“We have to establish better trust with the police and the citizens. When the police are writing the reports, they have to be accurate in telling the truth of what’s happening when it gets to the top channel. What I feel is going on is the citizens are not getting the true representation because some of this information is false, based on my own personal experience of what I went through,” he says. “I’m not scared for myself; I’m scared for the next kid that gets caught that doesn’t have Instagram, that doesn’t have the instant Twitter that can go viral.”

The hip-hop star is currently working with his legal team and deciding on whether he will sue the L.A. Sheriff’s Dept. but insists, “We’re not telling people ‘F the Police’ because if I say ‘F the Police,’ that’s like saying ‘F my cousin.’ I have police officers in my family. This is not a right issue or a left issue, this is an issue of the judicial system — the idea of putting laws in place that could actually protect citizens.”

Wednesday evening, the American Civil Liberties Union released a statement regarding the incident.

“The detention of internationally known musician and activist Wyclef Jean by Los Angeles Sheriff’s deputies reminds us once again that racial bias in policing continues to be a concern in California,” the ACLU’s statement reads in part. “This incident isn’t unique. The data we have shows that communities of color, and Black men in particular, are more likely to be stopped, pulled out of cars, frisked and searched, far beyond what is justified by actual criminal activity. These stops are more than just inconveniences. They threaten the dignity of those residents stopped, too often threaten their safety, and undermine already fragile trust in law enforcement.

“We call on the Los Angeles Sheriff’s Department to fully investigate the incident and outline how they plan to implement the Racial and Identity Profiling Act to ensure that incidents like this become a thing of the department’s past.”

—With additional reporting by CHAR ADAMS