Prince Andrew and Ex-Wife Fergie Visit Queen at Balmoral After Lawsuit Filed by Virginia Giuffre

"If I did not pursue this action, I would be letting [my family] and victims everywhere down," Virginia Giuffre said of why she's suing Prince Andrew over alleged non-consensual sexual encounters

Prince Andrew. Photo: STEVE PARSONS/POOL/AFP via Getty

Prince Andrew and his ex-wife Sarah Ferguson were photographed heading to Balmoral Castle in Scotland on Tuesday, one day after Virginia Giuffre (née Roberts) filed a lawsuit Monday regarding alleged sexual abuse by the royal.

According to photos obtained by the Mail Online, Queen Elizabeth's son drove a Range Rover to Balmoral, the Scottish estate where the monarch is currently spending her summer break. Fergie, as the Duchess of York is known, sat in the backseat of the car.

Fergie, who has remained an amicable co-parent and friend to the prince since their divorce in 1996, recently told PEOPLE of Andrew, "Whatever challenges he has, I will stand firm to the co-parenters we are together. I believe he's a kind, good man, and he's been a fabulous father to [Princess Beatrice and Princess Eugenie]."

For more on Prince Andrew being sued by Virginia Giuffre and other top stories, listen below to our daily podcast PEOPLE Every Day.

Giuffre has accused the royal of forcing her to have sex with him three times between 1999 and 2002 — accusations Prince Andrew has said could not be true.

Prince Andrew; Virginia Roberts Giuffre. Alexander Koerner/Getty; Emily Michot/Miami Herald/Tribune News Service via Getty

In a statement shared with PEOPLE on Monday, Giuffre said, "Today my attorney filed suit against Prince Andrew for sexual abuse under the Child Victims Act. As the suit lays out in detail, I was trafficked to him and sexually abused by him."

"I am holding Prince Andrew accountable for what he did to me," the statement continued. "The powerful and rich are not exempt from being held responsible for their actions. I hope that other victims will see that it is possible not to live in silence and fear, but to reclaim one's life by speaking out and demanding justice."

Spokespeople for Prince Andrew said they had no comment when reached by PEOPLE.

Prince Andrew. Alastair Grant/WPA Pool/Getty Images

Giuffre's statement concluded, "I did not come to this decision lightly. As a mother and a wife, my family comes first — and I know that this action will subject me to further attacks by Prince Andrew and his surrogates — but I knew if I did not pursue this action, I would be letting them and victims everywhere down."

Sigrid McCawley, managing partner at Boies Schiller Flexner who is representing Giuffre, told PEOPLE on Monday that Giuffre's case proves no one is above the law.

"The filing of this complaint proves that irrespective of power, privilege or even being a prince, no one is above the law in the United States and all perpetrators of abuse should be held accountable," said McCawley.

In response to earlier reports of the impending lawsuit, Giuffre's lawyer David Boies told the U.K.'s Mail on Sunday that Queen Elizabeth's son, 61, failed to respond to offers to settle her claims out of court.

Now that a civil action has been filed, Prince Andrew may be forced to be questioned under oath and turn over texts, emails and private letters related to the case.

Queen Elizabeth and Prince Andrew. Chris Jackson/Getty

In a 2019 BBC interview addressing his relationship with disgraced financier Jeffrey Epstein and Giuffre's allegations, Andrew said, "I have no recollection of ever meeting this lady, none whatsoever. It just never happened."

Giuffre alleges she was forced to have sex with the royal three times between 1999 and 2002 in London, New York and on a private Caribbean island owned by the Epstein, who died in in prison in August 2019.

Speaking specifically about Giuffre's claim that he danced with her at London's exclusive Tramp nightclub before allegedly having sex in a mansion in the upmarket Belgravia neighborhood, Queen Elizabeth's second son said, "No, that couldn't have happened because the date that's being suggested I was at home with the children."

Andrew added, "I'd taken Beatrice to a Pizza Express in Woking for a party at I suppose sort of four or five in the afternoon. And then because the Duchess [Sarah Ferguson] was away, we have a simple rule in the family that when one is away the other one is there. I was on terminal leave at the time from the Royal Navy so, therefore, I was at home."

When BBC Newsnight anchor Emily Maitlis raised the issue of a photograph showing the royal with his arm around Giuffre's waist — thought to have been taken inside the London home of Ghislaine Maxwell on March 10, 2001, when Giuffre was 17 years old — the royal claimed he had no recollection of the encounter.

Following the interview's backlash, Prince Andrew announced that he would be "stepping back" from public duties. He was most recently seen at the funeral of his father, Prince Philip, in April.