"I felt a little more protective of my body this time around," says Hilary Duff, reflecting on how her most recent pregnancy differed from the previous two

Hilary Duff wasn't down for sex during her pregnancy this time around, despite being all for it last time.

The Younger actress, 33, catches up with Dr. Elliot Berlin for a post-birth edition of Informed Pregnancy Podcast, reflecting on how her third pregnancy differed from the last two. Duff and husband Matthew Koma welcomed their second baby together, daughter Mae James on March 24. The couple also share daughter Banks Violet, 2, and Duff is also mom to son Luca Cruz, 9.

One contrasting point was that Duff didn't use sex during her pregnancy like a "helpful-slash-fun tool" like she did while expecting Banks.

"Sex really was not interesting for me at all this time around," she shares. "So toward the end, where with Banks that felt like a really helpful-slash-fun tool, this time I was like, 'Nope, not for me.' It annoyed me majorly when people would say, 'You know what really gets things started?' I'm like, 'Nope!' "

"Poor Matt," jokes Duff, "but maybe not. Maybe he was like, 'This is for the better right now.' "

She thought her reduced sex drive signified she was having a boy, Duff explains.

"I thought I was having a boy because with Banks I was totally down, and with this baby, I was like ... I don't know if it was maybe having more kids and trying to keep up with everybody that just made it kind of at night feeling exhausted and just really not a ton of time," says the mom of three. "I felt like a little more protective of my body this time around."

Want to get the biggest stories from PEOPLE every weekday? Subscribe to our new podcast, PEOPLE Every Day, to get the essential celebrity, entertainment and human interest news stories Monday through Friday.

Duff recounts her birth story, sharing how her labor and delivery happened very quickly — three hours and nine minutes in total.

"I was laughing at the way it was going down, because people were having to scramble to get home. I thought that was funny and I love to play tricks on people," she says. "There was humor in it and I felt really ready. Then it was probably about an hour and a half to two hours in, where they started to get super heavy and I was struggling to keep up. But my head was in the game and I felt a lot more confident in the way that I was birthing than I did before. So I was really trying to be present, and relaxed, and make every contraction work for me."

She later adds that she ultimately "gave one big, growling push and the baby was out" during her at-home water birth. Duff says, "I feel great. I think this has been my easiest one, my easiest recovery."

Duff and Koma, a singer/songwriter, announced their engagement in May 2019, and they tied the knot later that year in December. For their one-year wedding anniversary this past winter, Duff shared a throwback photo on Instagram from their nuptials and reflected on their marriage.

"Matt and I were both too nervous to eat the day of our wedding .. this is directly after ....we slammed a plate of apps someone brought us before taking pictures and this pretty much sums us up," she wrote. "Snacking through life, dreaming about dinner at 7 am and wrangling kids. Oh, and dodging covid the first year of marriage 😕. Always with you babe. For you. Over and over I choose you and this ♥️ thank you for being a damn good man."