Prince Charles and his wife Camilla, Duchess of Cornwall had some uncomfortable reading over breakfast at Balmoral Castle Monday morning.

As the 20th anniversary of the death of his ex-wife Princess Diana looms in 10 days, Charles was reminded that the negative legacy of that awful period is still being felt.

A poll by British national news agency the Press Association found that 27 percent of Britons felt Charles had a negative impact on the royal family – up from 15 percent in 2013. And only 36 percent say has been beneficial to the monarchy, compared to 60 percent four years ago.

F:PHOTOMediaFactory ActionsRequests DropBox47584#Getty ImagesGettyImages-52101541_master.jpg
Credit: Tim Graham/Getty

And his hope of having Camilla accepted as his Queen also took a knock. A mere 14 percent would like her to have that title when Charles is crowned King, with 39 percent believing she should be Princess Consort instead.

Camilla’s role in the royal marriage’s difficulties and the pain that caused Diana have been raised again as the princess’s life is celebrated and remembered in a series of films and documentaries such as PEOPLE’s The Story of Diana and books, like the re-release of Diana: Her True Story, by Andrew Morton.

FROM PEN: What’s Next for Princess Charlotte

Meanwhile, Diana’s chief of staff at the time of her death, Michael Gibbons, said in a rare interview that people would protest if Camilla was called Queen. “Whether Camilla will be consort or queen – I would think there’d be a riot if they suggested she should be Queen,” Michael Gibbins told The Sun.

“There’s been a lot of forgiving going on but there is still deep-seated public disquiet over the way she behaved.”

It came as the royals gather with Queen Elizabeth at Balmoral for the summer break. The family was together at the Scottish castle on the weekend that Diana was in Paris and suffered fatal injuries in the car crash.

William and Princess Kate headed to church on Sunday, joining the Queen and Prince Philip and Charles and Camilla and many of the rest of the family.