Prince Harry Says Visit with Queen Was Canceled After He and Meghan Markle Announced Step Back from Roles

In a new exclusive clip from Oprah Winfrey's sit-down with Prince Harry and Meghan Markle, the Duke of Sussex said that the Queen receives "bad" advice from her advisors

Prince Harry asserted that his grandmother Queen Elizabeth sometimes receives "really bad" guidance from her advisors, including an instance when she abruptly canceled a planned visit with the Duke and Duchess of Sussex.

In a new clip from Harry and wife Meghan Markle's bombshell Sunday interview with Oprah Winfrey, which aired on CBS This Morning, Monday, the prince said that after the couple finalized plans to step back as senior members of the royal family, Queen Elizabeth reversed course on a gathering with her grandson.

For more on Prince Harry and Meghan Markle's interview with Oprah, listen below to the episode of PEOPLE Every Day.

"That announcement that we put out on the 8th of January in 2020," Harry told Oprah in the new clip. "The contents of that was put into a letter to the institution, to my father [Prince Charles], which was then shared at the end of December while we were in Canada."

He continued, "And to then, get back on the 6th, after my grandmother had said, 'The moment you land, come up to [Sandringham House estate], we'd love to have a chat. Come for tea, why don't you stay for dinner?' "

Prince Harry, Meghan Markle and Oprah Winfrey. Joe Pugliese/Harpo Productions

Things changed once the couple arrived back in the U.K., though, with Harry saying he received a message from the Queen's private secretary "basically saying, 'Please pass on to the Duke and Duchess of Sussex that he cannot come to Norfolk. The Queen is busy, she's busy all week.' "

"So I rang her from [their former U.K. home Frogmore Cottage], that night, and said, 'I was thinking about coming anyway, but I hear you're now busy.' And she said, 'Yes, there's something in my diary that I didn't know that I had,' " continued the prince. "I said, 'Well, what about the rest of the week?' She goes, 'Oh, that's busy now as well.' Okay, I didn't want to push because I kind of knew what was going on.' "

Oprah then asked the pair, "Doesn't the Queen get to do what the Queen wants to do?"

Queen Elizabeth, Meghan Markle and Prince Harry. Max Mumby/Indigo/Getty Images

"No," said Harry, after some hesitation. "When you're head of the firm, there are people around you that give you advice, and what has also made me really sad is that some of that advice has been really bad."

Can't get enough of PEOPLE's Royals coverage? Sign up for our free Royals newsletter to get the latest updates on Kate Middleton, Meghan Markle and more!

Meghan and Prince Harry announced in Jan. 2020 they intended to "step back as senior members of the Royal Family" and split their time between the U.K. and North America.

"After many months of reflection and internal discussions, we have chosen to make a transition this year in starting to carve out a progressive new role within this institution," they said in a statement at the time. "We intend to step back as 'senior' members of the Royal Family and work to become financially independent, while continuing to fully support Her Majesty The Queen."

Get the premiere issue of PEOPLE Royals for glamorous new photos and inside stories royals fans haven't seen or read elsewhere! Subscribe at peopleroyals.com/launch

During the interview, Harry and Meghan also denied reports that the Queen and his father were blindsided by their announcement that they'd be stepping back.

"No, I never blindsided my grandmother," Harry said. "I have too much respect for her."

Discussing the interview on Monday, Oprah reflected, "That was such an important story to be shared, last night. I think the stories about blindsiding the Queen were very, very damaging to them and hurtful. ... There had been months and months of preparation."