The View's Whoopi Goldberg and Sunny Hostin criticized B. Smith's husband for opening up about his new girlfriend

Dan Gasby is in love with his girlfriend—but he also loves his wife, Barbara “B.” Smith.

Smith, 69, the 1970s cover model who became a successful restaurant owner, cookbook author and television personality, was diagnosed with Alzheimer’s in 2013. She suffered symptoms of the disease years before that and is now, as step-daughter and caregiver Dana Gasby, 32, tells PEOPLE in this week’s issue, a “physically strong toddler who speaks in gibberish,” who sometimes needs to be forced to shower and use the toilet.

Dan, 64, his wife’s other caregiver (they have no outside help), says he struggled with loneliness and depression as Smith’s Alzheimer’s progressed.

“I was tired. I was miserable,” he tells PEOPLE. “I could have put my wife in an institution, but I love her. It’s just a different type of love now.”

After Dan went public with his relationship with Alex Lerner, 53, whom he credits with saving him emotionally—and who helps with Smith’s care—he’s received death threats. He’s also been accused by strangers on social media of betraying the vows of his 26-year marriage and his race. (Lerner, a vegan chef and radio host, is white.)

Dan Gasby and B. Smith
| Credit: Andres Kudacki

The View’s Whoopi Goldberg used her talk show to castigate Dan for allowing The Washington Post to take video of Smith in her current condition. “I didn’t want to see B look like that,” said Goldberg. “I didn’t need to see that.”

And The View co-host Sunny Hostin, who said she is a friend of Smith’s and Dan’s, criticized Lerner for having her own bedroom in Smith’s East Hampton home, which Lerner visits on weekends.

“I find it very disrespectful that he is with his wife and disrespecting her by being with his girlfriend in their home,” Hostin said on air.

©mixmediaplus-2018-08-02 19.52.32-1_preview
Alex Lerner and Dan Gasby
| Credit: Danielle Franz/Mix Media Plus

To read more on Gasby, Smith, and Lerner, pick up the latest issue of PEOPLE, on newsstands Friday.

Dana, Dan’s daughter from a previous marriage and raised since the age of 5 by Smith, is having none of it. (Dan and Alex are scheduled to appear on The View on Thursday, according to an ABC spokesperson.)

“These fraudulent friends know nothing. I have not seen them here,” says Dana, who moved back home three years ago to care for Smith. “They don’t call my family. I see them talking on TV and how do you know what your supposed ‘friend’ would want?”

Dana, who fully supports the relationship between her father and Alex, says she hears Goldberg’s criticism of the video showing Smith’s day-to-day reality and wonders: is it “because you’re getting older and not happy with your health or whatever and you decide it’s not ok?”

Adds Dana, “Guess what—this is what aging is.”