Meghan Markle and Kate Middleton Are 'Not the Best of Friends' but 'Got On' Last Christmas

"It really was a special time for all of them to spend together," a source tells PEOPLE

Holidays can be a stressful time for families — royals included.

As the Cambridges and Sussexes head into Christmas, claims have swirled about tension between brothers Prince William and Prince Harry and their wives, Kate and Meghan.

But contrary to those reports, a source tells PEOPLE that the couples enjoyed a memorable first Christmas together last year.

“They really loved Christmas and all had a wonderful time,” the source says. “I think the two women got on — they are definitely not best friends, but it was a really special time for them all to spend together.”

Theimagedirect.com

William, 36, and Harry, 34, are bonded of course not just by blood but by a lifetime of growing up together on the public stage — and weathering the tragic 1997 death of their mother, Princess Diana. There is no doubt that they are increasingly moving apart in adulthood, as Harry and Meghan relocate to a new home in Windsor and the two couples separate their business offices.

The Image Direct

Many insiders were surprised that Meghan and Harry didn’t take over Apartment 1 (which has around 20 rooms) at Kensington Palace from the Queen’s cousins, the Gloucesters, but that has been said to require considerable overhaul to transform it into a modern family home.

• Can’t get enough of PEOPLE’s Royals coverage? Sign up for our newsletter to get the latest updates on Kate Middleton, Meghan Markle and more!

The move into Frogmore Cottage — a five-minute walk from Windsor Castle — will at least provide a peaceful place (apart from airplanes overhead landing and taking off from Heathrow nearby) to raise children and plenty of space to walk their two dogs. The “goldfish bowl” of Kensington Palace doesn’t provide that same escape.

Kate Middleton Opens Up About Meghan Markle’s Baby News for the First Time: ‘It’ll Be Special’

Those who know the princes say that there was always a chance that they would not want to live in each other’s backyards.

“Harry will want to establish himself and come out of the shadows and not be in his brother’s shadows,” says one insider. “They want to do their own thing and create their own style. And to do that you need to be away sometimes.”