"We were very keen to keep Prince Harry, he's been fantastic," a Rugby Football League source reportedly told the Telegraph

The Rugby Football League will be missing their beloved patron.

Buckingham Palace confirmed in a statement on Friday that Meghan Markle and Prince Harry — who left full-time royal work in March 2020 — will officially not be returning to their royal roles. As part of stepping down, the couple will no longer keep their patronages (their royal involvement with numerous U.K. charities).

Trusts and patronages will now revert back to Queen Elizabeth, 94, who will then distribute them among other working royals.

For more on the Royals, listen below to the episode of PEOPLE Every Day!

Two of those patronages were the Rugby Football Union and the Rugby Football League, which may see Prince William stepping in the vacant role, according to a report from The Telegraph.

Prince Harry, Duke of Sussex
Prince Harry
| Credit: Tim Rooke/Shutterstock

"There will be a lot of disruption. The World Cup is in autumn and Prince Harry was expected to be front and center of that," a source from the Rugby Football League reportedly told the outlet, which also reported: "There have been rumors that he could have some role, that there could be some middle ground, but whether that is feasible I don't know."

The Rugby Football League source added, "We were completely blindsided by the news to be honest. We were very keen to keep Prince Harry, he's been fantastic."

The World Cup is expected to take place in northern England.

"The Duke of Sussex was a high-profile figure who embraced the World Cup and we hope the next appointment will be a continuation of that," the Rugby Football League source told the outlet.

A spokesperson for the Rugby Football Union did not immediately respond to PEOPLE's request for comment.

Can't get enough of PEOPLE's Royals coverage? Sign up for our free Royals newsletter to get the latest updates on Kate Middleton, Meghan Markle and more!

Among the other trusts and patronages, Meghan and Harry are losing include the Queen's Commonwealth Trust, the Royal National Theatre, and the Association of Commonwealth Universities.

In addition, Harry will also lose his honorary military appointments.

In their own statement about the news, Harry and Meghan's office said that the pair "remain committed to their duty and service to the U.K. and around the world, and have offered their continued support to the organizations they have represented regardless of official role."

Several of the patronages shared their gratitude to the Duke and Duchess of Sussex on social media amid the announcement. The Rugby Football Union and the Rugby Football League made sure to thank Harry for serving as their patrons and shared photos of the Queen's grandson at some of their events over the years.

"We would like to thank Prince Harry for his time and commitment to the Rugby Football Union both in his position as Patron and Vice Patron," the Rugby Football Union said in a statement. "The RFU has greatly valued his contribution to promoting and supporting the game. We look forward to welcoming the Duke and Duchess of Sussex back to Twickenham at some point in the future."