'People Around the Queen Are Sad' amid Backlash from Prince Andrew's Role at Prince Philip's Memorial

Despite disquiet from royal family members, the Queen "insisted" Andrew accompany her to the service

Queen Elizabeth and Prince Andrew. Photo: RICHARD POHLE/POOL/AFP via Getty Images

A memorial service honoring Prince Philip was overshadowed by the image of Queen Elizabeth entering Westminster Abbey on the arm of their disgraced son, Prince Andrew, leaving those around the monarch "sad."

The Queen, 95, and Prince Andrew, 62, traveled by car together from Windsor to the Service of Thanksgiving in London, where Andrew escorted his mother down the aisle before they took their seats. The unexpected move came just six weeks after Andrew settled a sexual assault lawsuit with his accuser Virginia Giuffre.

"People around the Queen are sad that all everyone was talking about is Prince Andrew and not Prince Philip," a royal insider tells PEOPLE in this week's issue.

Queen Elizabeth and Prince Andrew. Richard Pohle/Getty

Royal commentator Robert Jobson says there was uneasiness among senior members of the royal family about the Queen's decision, "but she insisted."

"It shows she wholeheartedly loves and believes her son," Jobson, the author of Prince Philip's Century. "As she did when she made a statement about Camilla being Queen's Consort, many people will now accept the Queen's word and judgment."

"It does make some sense that he accompany her because he doesn't have a partner. A settlement has been paid but he's guilty of nothing in the eyes of the law," he adds. "She has faith in Andrew. Even if he disappears from public life, he's been able to pay tribute to his father, who after all, was very proud of his service in the Royal Navy, where he fought in the Falkland Islands conflict."

Queen Elizabeth and Prince Andrew. Chris Jackson/Getty Images

In January, the Queen stripped Prince Andrew of his military titles and patronages amid a sexual assault lawsuit filed against him by Giuffre, who alleges she was trafficked by convicted sex offender Jeffrey Epstein and forced to have sex with Prince Andrew at the age of 17. Andrew has denied any wrongdoing but announced in Nov. 2019 that he would "step back from public duties for the foreseeable future" following his bombshell interview with the BBC about his ties to Epstein.

The decision to strip Andrew of his titles and affiliations was "widely discussed" within the senior ranks of the royal family, a royal source confirms to PEOPLE — likely meaning talks between Queen Elizabeth, Prince Charles and Prince William.

Sarah Ferguson and Prince Andrew. getty

Prince Andrew's ex-wife Sarah Ferguson shared a series of Instagram posts on his behalf over the weekend using his banned "His Royal Highness" title — before deleting the message just hours later.

In the message, Prince Andrew reflected on the 40th anniversary of his leaving to serve in the Falkland War with the Royal Navy, where he flew missions as a Sea King helicopter pilot. The Queen's son said he entered the war "full of bravado" but returned a "changed man."

Shortly after it was posted, the Instagram message was amended to remove the "HRH" ahead of Prince Andrew's Duke of York title. Quickly thereafter, the posts were deleted entirely.