Windsor Castle is close to Harry and Meghan's new Frogmore Cottage home

Meghan Markle and Prince Harry may have moved away from Kensington Palace, but they still have royal family members nearby at their new home in Windsor.

Queen Elizabeth and Prince Philip often stay at Windsor Castle, which is one of the monarch’s favorite royal residences.

The historic castle is close to Harry and Meghan’s new Windsor home, Frogmore Cottage.

“When the Queen is there on a Sunday afternoon, it is a five-minute walk up the hill for tea with Granny. It is gorgeous,” a former palace staffer tells PEOPLE.

With its lush gardens, green fields and serene lake, the grounds of Windsor offers endless acres to roam with the couple’s two dogs and new baby.

100th Anniversary of the Royal Air Force, Westminster Abbey, London, UK - 10 Jul 2018
Queen Elizabeth, Meghan Markle and Prince Harry
| Credit: David Fisher/REX/Shutterstock

“It has the most amazing mulberry walk, where we would pick mulberries for Prince Philip’s mulberry gin,” the staffer adds.

Frogmore Cottage
| Credit: The Image Direct

The cottage is also just a short walk to the charming High Street, where local vendors sell everything from fresh flowers to baby clothes, while beef from the specialized pedigree Sussex herd that grazes nearby is available from the farm shop in the opposite direction.

In typical English style the shopkeepers are stiff upper-lipped about their new arrivals.

The Duke And Duchess Of Sussex Attend A Commonwealth Day Youth Event At Canada House
Meghan Markle and Prince Harry
| Credit: Samir Hussein/WireImage

“Outsiders may say ‘Wow’ when you say that the Queen has been shopping here,” says Martyn Crossley, proprietor of the local flower shop, who has created bouquets for Kate Middleton, the Queen and other members of the royal family. “But when you have been here 25 to 30 years it’s part of living in Windsor. And we have had fires and divorces and other dramas!”

  • Can’t get enough of PEOPLE’s Royals coverage? Sign up for our newsletter to get the latest updates on Kate Middleton, Meghan Markle and more!

Meghan and Harry, of course, are looking forward to a lifetime of happiness in their new hometown.

Moving to Windsor “is a really healthy thing to do,” says a longtime friend, noting that the rigid constraints of Kensington Palace are not for all: “I presume it must be nice to get out and away. Without neighbors who are all either family or staff [at Kensington Palace], they will now have their own thing.”