The star reveals the texts he got from his daughters about steamy scenes

Credit: ABC

There’s no denying it. Fitz is fit.

And Shonda Rhimes is giving Scandal fans plenty to enjoy as Tony Goldwyn‘s character enjoys a slice of the single life.

It’s a story arc that seems to live by the simple credo, “No Shirt, No Shoes, No Problem.” But not all Gladiators were happy to see the impossibly ripped 55-year-old actor going from presidential to shirt-popping.

“They’re cool with it [now], because my girls are grown up, but it [bothered them] initially, because they like the show,” Goldwyn recently told reporters of his two daughters, Tess and Anna. “So they would give me a hard time about it. During the show they would be like, ‘that’s disgusting!'”

He added they were not shy about letting him know their thoughts.

“They would text me, ‘Eeewwwww! Gross!’ So that was fun.” he said, laughing. “But now maybe they get up and make a sandwich [during sex scenes] or something.”

RELATED VIDEO: Did You Know Bellamy Young Was Only Supposed to Be on Scandal For Three Episodes?!

While he recently joked on Jimmy Kimmel Live! that his character may as well join Tinder, Goldwyn revealed to PEOPLE the single motivating factor that keeps him in fling-ready form.

“Fear,” he deadpanned. “I have to stay in reasonably good shape so that I’m in striking distance of not being humiliated.”

Scandal airs Thursdays (9 p.m. ET) on ABC.