Early Monday morning, Kardashian West was robbed at gunpoint in her Paris apartment

Kanye West is increasing security for his wife Kim Kardashian West after she was held at gunpoint and robbed in her Paris apartment.

At 2:30 a.m. Paris time on Monday morning, the Keeping Up with the Kardashians star stayed back in her No Address Hotel room while her bodyguard Pascal Duvier was with her sisters Kourtney Kardashian and Kendall Jenner as they partied in Paris.

Just after Kardashian West FaceTimed with Rob Kardashian and Blac Chyna during the couple’s baby shower, five men disguised in police uniforms tied her up and robbed her of two cellphones and approximately $10 million worth of jewelry, including a $4 million ring. After they left, a source tells PEOPLE Kardashian was able to wriggle out of her tethers and alert police of the incident.

Meanwhile, West, 39, performed “Heartless” at the Meadows Music & Arts Festival before he found out about the terrifying heist. “I’m sorry. I’m sorry. Family emergency, I have to stop the show” he told the audience before abruptly leaving the stage.

Although a rep for the KUWTK star told PEOPLE following the burglary that “she is badly shaken but physically unharmed,” the rapper is taking proactive steps to increase the security surrounding his wife moving forward.

“Kanye is very angry about what happened,” a source tells PEOPLE. “He always loved the apartment and has had it for years. He will obviously be changing the security details from now on. He feels crappy about what Kim had to go through.”

Kardashian West, 35, left France on Monday just hours after the incident and returned to New York City, where she reunited with West surrounded by throng of security personnel at her Manhattan apartment.

And though Kardashian West’s bodyguard Duvier was absent during the life-threatening incident, a second source close to the Kardashian family tells PEOPLE it’s unlikely he will be blamed.

“I cannot even remotely imagine Kim or the family is blaming Pascal. He is not only Kim’s security but also security for the family. He’s incredibly professional and takes his job with the family VERY seriously,” the second source tells PEOPLE. “Kim herself probably presumed things were safe in her room when Pascal was out with the group.”

With reporting by ELIZABETH LEONARD