Harry and William chatted together naturally as the family exited the chapel following the service

After more than a year apart, Prince Harry and Prince William came together to honor their grandfather.

The world was watching as the brothers reunited at Prince Philip's funeral on Saturday. The two walked in the procession behind their grandfather's coffin separated by their close cousin Peter Phillips, but William and Harry chatted together casually as the family exited the chapel following the service.

"The procession choreography, in retrospect, was a mistake," historian Robert Lacey, author of the bestselling biography Battle of Brothers, tells PEOPLE in this week's cover story. "As we saw, they could have walked side-by-side quite happily after all. What pleased me afterward was that it all seemed so totally natural, and they drifted to each other like the old days."

Harry, 36, and William, 38, were seen chatting alongside William's wife Kate Middleton as they left St. George's Chapel in Windsor Castle. The siblings continued speaking as they walked away from the church, as Kate stepped back to talk with Sophie, Countess of Wessex.

The Funeral Of Prince Philip, Duke Of Edinburgh Is Held In Windsor
Prince William and Prince Harry with Peter Phillips at Prince Philip's funeral.
| Credit: Alastair Grant/WPA Pool/Getty Images

Prince William and Prince Harry not walking together in the funeral procession or into the church was "a practical change rather than sending a signal," a Buckingham Palace spokesman said ahead of the funeral. "This is a funeral and we are not going to be drawn into perceptions of drama. The arrangements have been agreed and represent Her Majesty's wishes."

The Funeral Of Prince Philip, Duke Of Edinburgh Is Held In Windsor
Prince William, Peter Phillips and Prince Harry
| Credit: Samir Hussein - Pool/Wireimage

Harry returned to the U.K. from California for the first time in over a year to attend his grandfather's funeral following the Duke of Edinburgh's death at the age of 99 on April 9. Ahead of the service, Harry quarantined at his Frogmore Cottage home in the U.K.

Meghan Markle — who is expecting the couple's second child, a baby girl — did not accompany her husband. She "has been advised by her physician not to travel," a royal spokesman said.

prince philip, prince william, prince harry
Prince William, Queen Elizabeth, Prince Harry and Prince Philip
| Credit: Getty

Can't get enough of PEOPLE's Royals coverage? Sign up for our free Royals newsletter to get the latest updates on Kate Middleton, Meghan Markle and more! 

Despite tension within the royal family due to Prince Harry and Meghan's step down from royal life followed by their tell-all interview with Oprah Winfrey that aired last month, Prince Philip's death was a time to put family drama aside.

"It has been a very difficult time," a well-connected royal insider says of the past year. "The Duke was the head of the family, and if there were any time they were going to come together, it's now."