"As a family, they all love each other," a source tells PEOPLE

Meghan Markle and Prince Harry are getting in some quality time with Queen Elizabeth while they’re back in the U.K. this week.

The monarch, 93, invited the Duke and Duchess of Sussex to join her for church service in Windsor on Sunday morning.

“It was a really sweet gesture that the Queen asked them to church,” a source tells PEOPLE. “It’s telling in the sense that these two are still her family. And as a family, they all love each other.”

Meghan wore a fascinator-style headpiece and striking emerald earrings, while Harry wore a suit and tie. The Queen arrived in a light blue ensemble with a matching hat.

Prince Harry, Duke of Sussex and Meghan Markle, Duchess of Sussex
Credit: James Whatling
Queen Elizabeth II
Queen Elizabeth
| Credit: James Whatling

The move echoes the Queen’s rare and emotional statement earlier this year in which she said Meghan, Harry and her great-grandson Archie “will always be much loved members of my family,” following the announcement that Harry and Meghan are stepping down as senior working royals.

“Harry, Meghan and Archie will always be much loved members of my family,” the monarch said. “I recognise the challenges they have experienced as a result of intense scrutiny over the last two years and support their wish for a more independent life. I want to thank them for all their dedicated work across this country, the Commonwealth and beyond, and am particularly proud of how Meghan has so quickly become one of the family. It is my whole family’s hope that today’s agreement allows them to start building a happy and peaceful new life.”

Queen Elizabeth II
Queen Elizabeth
| Credit: James Whatling

Meghan and Harry are staying at their Frogmore Cottage home in Windsor while they are back for their final round of royal engagements this week. They made the quick trip up the road to join the Queen for the Sunday service.

Harry, who has always had a close relationship with his grandmother, also had a private lunch with the monarch at Windsor Castle last Sunday before Meghan arrived in the U.K. from Vancouver Island in Canada, where she, Harry and Archie have been staying in recent months.

Royal sources deny that the meeting was a top level summit to discuss his royal exit, and that it was more along the lines of a grandson having Sunday lunch with his grandmother.

Sussex Royals
Meghan Markle, Prince Harry and Queen Elizabeth
| Credit: John Stillwell - WPA Pool/Getty

It was the Queen’s experience with her younger sister Princess Margaret that influenced her decision to let Harry set out on his own.

“She has a particular sensitivity to what Harry has been going through because of her closeness to Margaret and seeing the same dramas and tensions played out two generations ago,” royal biographer Robert Lacey tells PEOPLE.

Harry and Meghan’s final royal engagement will be on Monday when they join the Queen and other senior members of the royal family, including Prince William and Kate Middleton, for a service at Westminster Abbey to mark Commonwealth Day.