Duff Goldman Accidentally Unmasked Himself on 'The Masked Singer' : 'I Was Really Worried'

"I didn't want to ruin the show," Food Network star Duff Goldman tells PEOPLE about what was going through his head during his snafu on the season 7 premiere of The Masked Singer

Duff Goldman had a ruff turn on The Masked Singer.

But the 47-year-old Food Network star — who competed as McTerrier, the Scottish terrier — didn't really mind being the first one eliminated from season 7 on Wednesday.

"I wasn't disappointed," Goldman tells PEOPLE exclusively, of how he felt about leaving the show. "I was just bummed that I wasn't able to sing my next song because I was really excited about it."

The Spring Baking Championship judge reveals he planned to sing "All Star" by Smash Mouth if he moved on, and then would've liked to perform "Take Me Home, Country Roads" by John Denver. "I love West Virginia," Goldman says. "It's one of my favorite places in the country."

But Goldman rocked out pretty hard during his one and only performance — so much so that he lost his McTerrier head after singing "Working for the Weekend" by Loverboy. Luckily, judges Ken Jeong, Jenny McCarthy, Nicole Scherzinger and Robin Thicke turned their backs and avoided identifying the father of 13-month-old daughter Josephine before he ultimately got voted out.

Below, Goldman opens up about being an IRL rocker, how much he could really see under the costume and which of his Food Network colleagues would most likely appear on The Masked Singer.

Michael Loccisano/Getty; Michael Becker/FOX

PEOPLE: Why did you want to go on The Masked Singer?
Goldman: It's very outside of my comfort zone. Usually, when I'm performing, I'm baking cookies. So I thought it'd be totally different, a really fun new thing. I love music. Music's such a huge part of my life. Getting to go out there and celebrate that, it was great. I would do it again in a heartbeat.

You opened your performance by playing drums. How long have you played music?
I started playing music when I was about 6, but I started being in bands when I was in college. And I've been in touring rock bands. I've been in cover bands. I'm actually in a band right now that's all chefs. It's called Foie Grock. Everybody else that works at Charm City Cakes, we all either are in bands or somehow are involved with music and performance.

Why did you select the McTerrier outfit?
Have you ever seen Best in Show, the movie?

Yes.
So remember Winky the Norwich Terrier? He wins the whole show. And he was like — no pun intended — he was such an underdog. I felt like I was definitely an underdog here. But if Winky could do it, I could do it.

Michael Becker/FOX

You really committed to the Scottish accent.
They put me in there and they were like, "Hey, can you do a Scottish accent?" I was like, "No. I don't think so." So I watched a bunch of Mike Myers movies. And there's this guy Billy Connolly, and back in the '80s, he had a really funny stand-up special on HBO. So I found it on YouTube, and I watched that and kind of listened to his accent. So I was trying to do it. I don't know if I fooled anyone, though. I don't think anybody actually thought I was Scottish.

Ken thought you were having a really tough time seeing under there. How much could you actually make out?
You can't see a thing. And it's hot. God, it's hot. You can kind of see a little bit out the mouth, but when I was playing the drums, I was totally blind. I played them a few times just so I could feel where everything was, like where the toms were, where the cymbals were. So when the song started, I was like, "All right. I hope I'm centered." You can't see anything. That was like a five-foot drop. I did not want to fall off that stage. That was terrifying.

So it's safe to say you can't bake in there.
It would be super dangerous.

What was going through your mind after you lost your costume head?
I was just really worried that I let everybody down. I didn't want to ruin the show. It's this big huge Fox production and I was like, "Man, I don't want to be the one that just completely ruins this episode," and like, "Oh great, now we can't use him because his head fell off." That was my biggest worry.

Duff Goldman. Discovery+

As someone who typically does the judging on Food Network, how did it feel to be the one being judged this time?
When I started on Food Network, I was doing all these cake decorating competitions, so I've been a competitor on TV shows for years now. The biggest thing that I think is important, when you're being judged for something as fun and silly as Masked Singer or something as "serious" as a baking competition, is 1) you take everything with a grain of salt, and 2) none of it's personal.

That's great advice!
Yeah, as a judge on shows, people don't realize is I'm really pulling for people. I want them to do well. So when I critique, usually what I'm doing is I'm trying to give them hints to help them win. I'm not saying, "I didn't like this brownie. It was dry." Like, "Okay, the brownie was dry. Here's how to avoid that in future." All these [Masked Singer] judges, they weren't really critiquing my singing performance. Nobody thought I was actually a professional singer. We were just all having a lot of fun with it.

Do you think any of your Food Network colleagues would appear on The Masked Singer?
Probably Guy [Fieri]. Guy likes to be super famous, so he is probably like, "Wait, Duff is on Masked Singer? I want to be on Masked Singer."

courtesy Duff Goldman

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Did your daughter recognize you on TV after the reveal?
Whenever she sees a dog, she says, "Waf, waf! Waf, waf!" It's really cute. I do a puppet show — it's on Discovery+, called Duff's Happy Fun Bake Time — and it's me with a bunch of puppets. We let her watch that show. And when I came out she says, "Dada! Dada!" So she knows, even if I'm on TV, she's been able to say "Dada." She's so smart. It's amazing. She said "food" the other night.

The Masked Singer airs Wednesdays (8 p.m.) ET on Fox.