Prince Harry Says Prince William Screamed and Shouted at Him Over Decision to Step Back from Royal Life

In Harry & Meghan, Prince Harry recalled when he met with Queen Elizabeth, Prince Charles and Prince William at the Sandringham Summit to talk about his and Meghan Markle's exit from their royal roles

Prince Harry and Meghan Markle are sharing new details about the Sandringham Summit.

In the second volume of their Netflix docuseries Harry & Meghan, the Duke and Duchess of Sussex opened up about the meeting called by the late Queen Elizabeth after they announced their intention to step back as senior working members of the royal family in January 2020.

"I sent an email to the three most senior private secretaries saying, 'Let's have a meeting, let's get together and have a meeting and talk about this,' " Harry said. "Because what was happening, what was playing out in public was crazy. And that meeting was rejected."

For more on the Royals, listen below to our daily podcast PEOPLE Every Day.

"It was only once Meg had left and gone back to Canada that it was then arranged that there was going to be a meeting at Sandringham on the following Monday," he continued.

"Imagine a conversation, a roundtable discussion about the future of your life," said Meghan. "When the stakes are this high. And you as the mom and the wife and the target, in many regards, aren't invited to have a seat at the table."

Chris Jackson/Getty

"It was clear to me that they planned out so that you weren't in the room," added Harry.

Prince Harry then revealed the inside events of his approximately 90-minute discussion with Queen Elizabeth, the then-Prince Charles and Prince William at the monarch's country estate in Norfolk to "talk things through" on Jan. 13, 2020 — five days after Meghan and Harry released their statement.

Prince William and Prince Harry. Alamy Stock Photo

"I have such happy memories of Sandringham — it was where we spent every Christmas — and now I was back there in very different circumstances," he said about his fondness for the royal family's special annual gathering. During her life, Queen Elizabeth would travel to Norfolk in mid-December and stay there until the anniversary of her father's death and her accession to the throne in early February.

"I went in with the same proposal that we'd already made publicly. But once I got there, I was given five options," added Harry. "One being 'all in, no change,' five being 'all out'."

"I chose option three in the meeting," he said. "Half in, half out; have our own jobs but also work in support of the Queen."

Despite this, he said that his decision provoked a reaction from the family that was not one of compromise — and led to his older brother, Prince William, shouting at him.

"It became very clear very quickly that that goal was not up for discussion or debate," said Harry. "It was terrifying to have my brother scream and shout at me, and my father say things that simply weren't true and my grandmother quietly sit there and sort of take it all in."

Sandringham House. Indigo/Getty

"But you have to understand that, from the family's perspective, especially from hers [Queen Elizabeth], there are ways of doing things — and her ultimate sort of mission goal or responsibility is the institution. People around her telling her, by the way, that proposal or these two doing 'xyz' is going to be seen as an attack on the institution, then she's going to on the advice that she's given," he continued. "That was really hard."

According to Prince Harry, the meeting "finished without any solidified action plan."

"I think from their perspective, they had to believe that it was more about us and maybe the issues that we had, as opposed to their partner, the media, and themselves. That relationship that was causing so much pain for us. They saw what they wanted to see," the Duke of Sussex said.

"The saddest part of it was this wedge created between myself and my brother, so that he's now on the institution's side," he continued. "Part of that I get, I understand, right, that's his inheritance. So to some extent, it's already ingrained in him that part of his responsibility is the survivability and continuation of this institution."

Prince William and Prince Harry. Chris Jackson/Getty Images

On the same day at the Sandringham Summit, the palace released a joint statement from Prince William and Prince Harry pushing back on a media report that the rift between the brothers was allegedly caused by William's "bullying." However, Harry said in the Netflix show that he did not play a part in writing the statement.

"We couldn't believe it," the Duke of Sussex said. "No one had asked me permission to put my name to a statement like that. And I rang M and I told her, and she burst into floods of tears because within four hours, they're happy to lie to protect my brother. And yet, for three years, they were never willing to tell the truth to protect us."

Meghan said in the docuseries, "Suddenly, what clicked in my head was 'it's never going to stop.' Every rumor, every negative thing, every lie, everything that I knew wasn't true and that the palace knew wasn't true and internally they knew wasn't true, that was just being allowed to fester."

Prince Harry, Queen Elizabeth and Meghan Markle at Royal Ascot in 2018. Max Mumby/Indigo/Getty Images

Can't get enough of PEOPLE's Royals coverage? Sign up for our free Royals newsletter to get the latest updates on Kate Middleton, Meghan Markle and more!

Shortly after the Sandringham Summit, Queen Elizabeth released a rare and emotional statement.

"Today my family had very constructive discussions on the future of my grandson and his family," she said. "My family and I are entirely supportive of Harry and Meghan's desire to create a new life as a young family," she continued. "Although we would have preferred them to remain full-time working Members of the Royal Family, we respect and understand their wish to live a more independent life as a family while remaining a valued part of my family."

"Harry and Meghan have made clear that they do not want to be reliant on public funds in their new lives. It has therefore been agreed that there will be a period of transition in which the Sussexes will spend time in Canada and the U.K.," she continued. "These are complex matters for my family to resolve, and there is some more work to be done, but I have asked for final decisions to be reached in the coming days."

In March 2020, Prince Harry and Meghan took part in their final round of royal outings before relocating to Meghan's home state of California. Almost a year later, in February 2021, Buckingham Palace confirmed that the couple would not be returning to their royal roles.