Mark Wahlberg Reveals His Family Went on Christmas Vacation Without Him When He Had COVID

"I’m down in the basement a couple days before Christmas, I don’t get to open any gifts with anyone or anything and next thing you know, they’re still going on holiday," says Mark Wahlberg

Mark Wahlberg's family was ready to take their Christmas vacation no matter what — even if it meant leaving Dad behind.

The actor, 50, appears on Thursday's episode of The Ellen DeGeneres Show where he shares the story of when his family went on Christmas vacation without him when he was stuck at home with COVID-19.

"You had COVID during Christmas and that must have been difficult because your family just went off and left you," says DeGeneres.

"We always go on holiday at Christmas and I'm down in the basement a couple days before Christmas, I don't get to open any gifts with anyone or anything and next thing you know, they're still going on holiday," Wahlberg shares.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Michael Rozman/Warner Bros.

"I hear the car getting ready to go, the doors are shutting, and I'm like, 'Are you not the gonna come to the stairs and say goodbye?' " he says with a laugh. "But they did come to the stairs and stayed not 6 feet apart but like 36 feet apart."

After receiving two negative tests, the Ted star was able to meet them on the trip but says his family wasn't too thrilled to see him.

"They're like, 'What are you doing here? Are you sure he's okay?' I still had a little bit of a cough and stuff like that. They didn't want me there," he jokes.

"I understand because the last holiday we had, my wife got COVID the first day we arrived and then my youngest son and my youngest daughter all got it and they were quarantined. And we had to leave to go to New York so it was a little payback, a little karma," he shares.