The reality star says he and wife Korie "always keep an open mind and heart" when it comes to growing their family

Credit: Marion Curtis/Startraks

When Duck Dynasty‘s Willie Robertson saw a tabloid proclaiming that he was adopting again, he had to pick up a copy.

“I always like to read what I’m up to,” he tells PEOPLE with a laugh. “[The tabloids] always seem to know what’s up with my life before I do.”

So what’s the truth? Does he really want to adopt again? “I will never say never,” he says firmly. “Adoption is something very close to my heart. We are not currently in the process of adoption, but I always keep an open mind and heart to it.”

Robertson, 41, and his wife, Korie, 40, adopted their son Will, 11, when he was just five weeks old – and have always been open with him about his past. “It’s never been a secret,” says Robertson. “For one thing, he’s biracial, so it was obvious. But here’s the thing: when we look at him, we don’t see an adopted kid. It doesn’t even cross our minds. He’s as much ours as our biological children.”

The Robertsons made the decision to be open about their family because they hope to raise awareness about adoption. “If we’re able to make even one family consider adoption, we’ll do it,” he says. “I don’t know where his life would have been – I won’t say it would have been bad – but he probably wouldn’t have been with a father if we hadn’t adopted him. There are so many kids out there who need help, and it doesn’t matter what age, color, sex they are.”

The Robertsons’ adoption was finalized around the holidays, which has added to the family’s love of Christmas. “A month after Will’s adoption, we found out we were pregnant,” he says. “We have always been blessed with the season. So we wanted to do something to share how we feel.”

The Robertsons, who are devout Christians, have always sung in church – but this year, they decided to sing to a wider audience. Collaborating with George Strait, Alison Kraus and Luke Bryan, they released a Christmas Album, appropriately titled Duck The Halls.

Yes, they’re wearing camouflage and standing in a swamp in publicity photos, but Robertson insists that the album is no joke. “Of course people are going to assume that this is a novelty album, but it’s much more than that,” he says. “There are some funny parts, but we do traditional Christmas songs as well. We really gave it our all.”

And what about rumors that they family isn’t getting along and is plagued by infighting? “At the end of the day, we’re always fine. We are a strong family,” says Robertson. “We’re not all alike, and we don’t always agree. We may have arguments and scream and fight. But we always come back together around the dinner table.”