Aside from adjusting to life as new parents, the Duke and Duchess of Sussex have had full schedules as the summer season comes to an end

By Stephanie Petit
September 06, 2019 10:00 AM

Meghan Markle and Prince Harry won’t be visiting Queen Elizabeth at her Scottish Highlands getaway this year.

Each summer, the 93-year-old monarch heads to Balmoral Castle in Scotland and invites royal family members to spend time with her at the private escape. However, it has emerged that the Duke and Duchess of Sussex won’t be heading there with 4-month-old son Archie this season.

Although Meghan has yet to visit the Scottish castle, a royal source says the couple see Prince Harry‘s grandmother regularly given their close proximity to her at Windsor Castle.

“When the Queen is there on a Sunday afternoon, it is a five-minute walk up the hill for tea with Granny. It is gorgeous,” a former palace staffer previously told PEOPLE after Meghan, 38, and Harry, 34, left London for Windsor’s Frogmore Cottage.

David Fisher/REX/Shutterstock

Aside from adjusting to life as new parents, Meghan and Harry have also had a busy summer. Later this month, they’ll head to Africa on a royal tour — their first as a family of three! — and Meghan has also been preparing for the unveiling of the capsule collection she has created with pal Misha Nonoo to benefit her patronage Smart Works. The clothing line’s launch next Thursday will mark the new mom’s official return to royal duties after her maternity leave.

But before heading back to work, Meghan has a personal trip underway. The Duchess of Sussex boarded a commercial flight from London on Friday morning, heading to New York City to see close pal Serena Williams in the US Open final this weekend. She is making the trip without Prince Harry and Archie, who will remain home in Windsor.

Meghan Markle, Archie and Prince Harry

Harry has also been busy launching his new eco-friendly tourism initiative, Travalyst, which he kicked off in Amsterdam earlier this week. The work trip away from Meghan and Archie had a personal benefit for Harry.

“Having spent last night here — I don’t know about you guys, but it was definitely the best night’s sleep I’ve had for the last 4 months!” he joked at the beginning of his speech.

Balmoral Castle
Giuseppe Masci/AGF/UIG/Getty Images

For more than 150 years, Balmoral has served as a private retreat for the British royals. In the highlands, the royals like to entertain friends, hold dances, picnic and go on shooting parties — all the while battling the nibbling little flies that stalk the heather-covered mountain sides.

It’s not all play for the monarch while she’s away in the Scottish Highlands. She does continue to work, reading her diplomatic papers in the famous “red boxes” that are delivered daily. She also holds audiences and undertakes some public engagements.

Queen Elizabeth and her family at Balmoral Castle in 1960
  • Can’t get enough of PEOPLE’s Royals coverage? Sign up for our newsletter to get the latest updates on Kate Middleton, Meghan Markle and more!

Kate Middleton and Prince William closed out their summer by bringing their three children — Prince George, 6, Princess Charlotte, 4, and Prince Louis, 1 — to Balmoral. The couple joined the Queen during her weekly outing to church during their time there, with William riding shotgun while Kate sat in the car’s backseat beside the monarch.