Royals

Inside the Palace: How the Royal Family Is Dealing with Prince Andrew's Scandal

Palace insiders tell PEOPLE it's business as usual for Andrew and the Queen – and that he has his family's support

Published on January 5, 2015 11:30AM EST

As Prince Andrew arrived back in the U.K. from his New Year’s break in the Swiss Alps, he was greeted by headlines about his alleged involvement in a persistent sex scandal.

“Andrew Tells the Queen: I’m Innocent” was the Daily Mirror’s take, while “Andrew Back to Face the Music” was the Daily Mail‘s.

And, as palace insiders tell PEOPLE, the ongoing Florida-based civil case that has now mired 54-year-old Andrew is expected to stick around for some time.

The claims, that he had sex with “Jane Doe 3” (now named by newspapers in the U.K. as Virginia Roberts) when she was 17 and a minor, stem from a motion filed in connection to a long-running lawsuit against billionaire investment banker Jeffrey Epstein. Epstein, a former friend of the prince’s, was convicted of soliciting sex with an underage girl following a plea bargain deal in 2008.

In the current civil suit, the woman claims Epstein “forced her” to have sexual relations with Andrew. The palace has called the allegation “categorically untrue.”

And, when newspapers repeated claims made by Roberts, a spokesman told PEOPLE Sunday that the palace “emphatically denied” that the prince had “any form of sexual contact or relationship” with her.

As Andrew settled back into Royal Lodge, his home in Windsor close to the famous castle, he started planning his next move with his senior advisers and lawyers.

“There is an acknowledgement that the civil case will last a considerable length of time. But he’s not going to go to ground,” a senior royal aide tells PEOPLE. “He will carry on what he has been doing in terms of his public engagements and work.”

Since Andrew has stepped down from his role as a U.K trade ambassador when his continuing friendship with Epstein became known in early 2011, he has refocused his public life around supporting economic growth and the creation of skilled jobs, especially in the worlds of science and technology.

One thing he won’t be doing yet is meeting his mother, Queen Elizabeth, face-to-face. Aides are not expecting a summit at Sandringham, where the Queen, 88, carried on as her usual stoic self as she went to church at St. Mary Magdalene’s Sunday. “He speaks with his mother frequently,” says a palace source without further comment.

“In 2011, just before William and Kate’s wedding, [Andrew] is said to have had a meeting with the Queen and he assured her there was no truth in the allegations,” says Majesty magazine’s editor-in-chief, Ingrid Seward. “Buckingham Palace’s decision to make these statements was probably pushed by him. They wouldn’t normally do this.”

Royal Family Support

The rest of the royal family remain supportive, a source says, while Andrew’s ex-wife, Sarah Ferguson, and his daughters, Princesses Beatrice and Eugenie, are wholeheartedly behind him.

“They’ve all had so many dramas, and they’ve stuck together,” Seward says of Andrew and Fergie. “He has stuck by her and she will stand by him.”

Meanwhile, Buckingham Palace denied that the Queen had met Roberts, as had been alleged by her father Sky in the Sun newspaper Monday. “There is nothing to suggest this is true. We have no record of this meeting,” a spokesman says.

With the marriage of Prince William and Kate, and the subsequent birth of Prince George securing the succession to the joy of the Queen, and a new baby on the way for the couple in April, it has been a period of celebration for the royals. Moreover, the Queen herself is set for a milestone in September when she will become the longest serving Monarch, surpassing Queen Victoria’s reign of 63 years and 216 days.

“Just as things were looking bright you get things like this coming along,” says Seward. “It’s detrimental to the monarchy. It casts doubt over the whole institution of the monarchy.”