Daphne Oz, Hilaria Baldwin and LaTonya Yvette Talk the Difficulty of Leaving Their Kids at Home

"Happy kids come from happy parents," Daphne Oz says on the newest episode of Mom Talk, also starring Hilaria Baldwin and LaTonya Yvette

Not even celebrities are immune to the infamous mom guilt.

“Right now, our kids want us to be there all the time, and I’m still at that place where I feel really guilty,” Hilaria Baldwin — pregnant wife of Alec Baldwin and mom to Leonardo “Leo” Ángel Charles, 18 months, Rafael Thomas, 2½, and Carmen Gabriela, 4½ — says in the latest segment of PEOPLE’s Mom Talk.

“I just feel really bad because then they get really upset,” explains the 34-year-old fitness guru of the times she leaves her children home to tend to other duties.

Want all the latest pregnancy and birth announcements, plus celebrity mom blogs? Click here to get those and more in the PEOPLE Babies newsletter.

Daphne Oz, LaTonya Yvette and Hilaria Baldwin

Watch Hilaria Baldwin talk parenting and more on Mom Talk. Go to PeopleTV.com, or download the PeopleTV app on your favorite streaming device.

“Well they know how to come and chase you and be like, ‘Don’t go, I want you to stay,’ ” pipes in Daphne Oz — mom to Domenica Celine, 3 months, Jovan Jr., 2, and Philomena Bijou, 4.

“Nothing will ever break your heart the way that little voice saying that [does],” adds Oz, 32. “As much as I tell myself, ‘Mom guilt helps no one,’ you just have to do what makes you happy — happy kids come from happy parents. They will feed off of that.”

Courtesy Daphne Oz

Daphne Oz and family

For more coverage, follow our babies magazine on Flipboard.

“It does get a little bit better,” says stylist and blogger LaTonya Yvette. “My daughter doesn’t blink … at 7, she’s like, ‘Okay, bye.’ ”

“My daughter says, ‘Mommy, I miss you even when I’m with you,” says Baldwin. “I’m like, ‘I better not go.’ Like, how much closer can you get?”