"No one close to him can tell him that he's sounding unhinged," a source close to Kanye West tells PEOPLE

By Steve Helling and Natalie Stone
October 11, 2018 07:10 PM

Kanye West‘s family is very concerned for the rapper — but doesn’t know what to do or how to help him.

During a meeting with Donald Trump in the Oval Office on Thursday, West — who avidly supports and voted for the president in the 2016 election — claimed during a lengthy, televised discussion that he was “misdiagnosed” with bipolar disorder. (The rapper, 41, also recently revealed that he is “off medication” he had previously been taking to treat the disorder.)

According to a source close to West, people in his inner circle “are telling him that he needs to get back on his medication, that he’s not doing well, that he’s not making any sense.”

“Now he’s in the Oval Office, and he’s doing the same rant, and that’s going to validate his rants,” shares the source.

While family and close friends want to support the father of three, they don’t feel that they’re able to get through to him — in part because of POTUS’ validation.

“No one close to him can tell him that he’s sounding unhinged,” says the source, “because his answer is that the President of the United States doesn’t think so.”

Trump, Washington, USA - 11 Oct 2018
Donald Trump and Kanye West
| Credit: Evan Vucci/AP/REX/Shutterstock

Although the Kardashians have been an unwavering support system to him, the source says that “the entire family is at the end of their ropes, and stuff like this won’t help.”

“They’re trying to help him get things where they should be, and it’s going to be that much harder now,” the source adds. “It’s a sad day.”

President Trump Hosts Kanye West And Former Football Player Jim Brown At The White House
Donald Trump and Kanye West
| Credit: Oliver Contreras - Pool/Getty

Over the course of the lengthy discussion with Trump, the hip-hop superstar touched on his mental health, the 2016 election and his famous in-laws, the Kardashians, albeit briefly. “I’m married to a family that..you know, not a lot of male energy going on there. But it’s beautiful, though!” he said, according to video obtained by CBS.

Earlier this week, West’s wife, Kim Kardashian West, shared her support for him ahead of his presidential meeting.

Speaking to ExtraTV at a Tiffany & Co. event in New York City on Tuesday, the Keeping Up with the Kardashians star said she’s “proud” of her husband.

Kim Kardashian West and Kanye West
| Credit: Kanye West/Twitter

Kardashian West, 37, revealed the meeting would be about “the city of Chicago and some exciting things that I feel like they have been planning, and I am really hopeful the president listens.”

As for whether Kardashian West planned to give him advice?

“I try to, I will try to help him communicate,” she told ExtraTV. “I always say he isn’t the best communicator, but he has the best heart. I know what he wants to accomplish … and I can’t wait to see that happen.”

The West Family
| Credit: Kim Kardashian West/Instagram

Although Kardashian West — she shares daughters North, 5, and Chicago, 8 months, and son Saint, 2½, with West — will stand by her man no matter what, his rants do take a toll on the reality star.

“Having all this backlash against Kanye is embarrassing for her and her family, but she will never say that,” a source previously shared.

Still, “she supports Kanye through and through and truly does think her husband is entitled to his own opinion.”