Eddie Redmayne is dad to son Luke, 4, and daughter Iris, 5
Eddie Redmayne
Credit: LIVEKellyandRyan/YouTube

Eddie Redmayne's little ones don't quite buy his real-life wizardry skills.

The Fantastic Beasts star, 40, appeared on Wednesday's episode of Live with Kelly and Ryan, where he chatted about his two kids and whether they are impressed by his magical character in the film series.

Redmayne, who shares son Luke, 4, and daughter Iris, 5, with wife Hannah Bagshawe, recalled two recent instances when his kids were unamused by their dad's role as wizard Newt Scamander.

"There was a moment this year where I felt like as a dad, who played a wizard, I could try and get some brownie points," he begins. "I gave Iris, my daughter, Newt's wand. Which I thought was pretty exciting, and she's like 'Okay, thanks Dad.' "

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

"I came back an hour later and she covered the thing in tinfoil, she'd made a little star and made a fairy wand. And she's like, 'Now this is a wand,' " Redmayne says with a laugh. "[That] slightly undermined my brownie points."

The actor says he's also struggled with explaining whether he's a real-life wizard when asked by his toddler son.

"You have that sort of moral dilemma, like do I lie to my child? I find myself stuck in the middle and he's like, 'No, you're either a wizard or you're not,' " Redmayne continues. "So I said 'Yeah I am a wizard' and I did some sort of trick with a coin, the one magic trick I could do."

"And he's like, 'That's not the kind of stuff you can do in the [Fantastic Beasts] trailer. In the trailer, you can make stuff explode,' " he says. "So I think he told his friends at school and they asked me to do some of the stuff in the trailer and it was a bit of a letdown."