Following their July 1981 wedding, Charles realized that the shy 20-year-old woman he had married was quickly adapting to her role in the spotlight — and growing in popularity with the public.
Diana became one of the most photographed (and talked-about) people in the world, and Charles felt overshadowed.
Lady Elizabeth Anson, party planner and a cousin to the Queen, tells PEOPLE, “[Diana] sort of metamorphosed into a very, very elegant blonde beauty. They did a visit to Wales which was the beginning of, for Prince Charles, realizing, ‘They’ve come out to see my wife, they haven’t come out to see me.'”
- Can’t get enough of PEOPLE’s Royals coverage? Sign up for our newsletter to get the latest updates on Kate Middleton, Meghan Markle and more!
Charles’ jealousy didn’t end with Diana’s fame — it continued with his eldest son’s marriage.
In his unauthorized biography of the Prince of Wales, Rebel Prince: The Power, Passion and Defiance of Prince Charles, author Tom Bower claims that Charles felt “usurped” by the Middletons and “isolated” from grandchildren Prince George and Princess Charlotte, when Prince William and wife Kate Middleton chose to spend Christmas with her parents rather than with the other royals and setting up their home base miles away from Highgrove (Charles’ residence) to Norfolk “where they could preserve their privacy.”
“Charles saw Kate and William as the new stars and feared he’d be in trouble,” Robert Higdon, the chief executive of Charles’ charity foundation in America, told Bower.
RELATED VIDEO: Prince Charles Was the ‘Architect of the Disaster’ of His Marriage to Diana
Anson is one of many sources who speak in depth about the Queen, her family, the past and future of the royal family in The Story of The Royals, a two-night television event presented by PEOPLE, airs August 22 and August 23 (9:00-11:00 p.m. EDT) on ABC.