Spencer shared the childhood memory of his late sister while honoring young leaders in her name
A Vision in White
Credit: Tim Graham/AP

Prince William and Prince Harry were hoping to catch up with their uncle Charles Spencer on Sunday, but family life intervened.

“One of them had childcare issues – you can imagine which one,” Spencer said at a charity event in his late sister’s name in London on Monday.

When it was suggested moving to today, he had to pass, saying he was giving out honors to young people from the charity set up in the name of their mother, Princess Diana.

“There was an absolute immediate applause from both of them for you,” he told the youngsters receiving The Diana Award on Monday. “So they wanted me to pass on their congratulations to you today.”

William and Harry have been big supporters of the charity, with William helping spearhead promotion of the charity’s anti-bullying drive.

Diana’s brother, who opens his Althorp home to PEOPLE in this week’s issue alongside his wife, Countess Karen Spencer, made a brief address before handing out awards to 80 young leaders nominated by the public – the the first direct member of her family to do so in Britain.

Want to keep up with the latest royals coverage? Click here to subscribe to the Royals Newsletter.

Calling his sister – who died in a car crash in Paris in 1997 and would have been 55 on July 1 – a “really inspirational person,” he told the audience in London that “when she died, one of the tragedies for me as her brother was I thought, ‘That will be that,’ and she will disappear into the background.”

But that has hardly been the case. “Various things were put forward as a commemoration of her, and all of them were lovely, but this is the best because it’s a whole generation of new people bearing her name and carrying on her work. Thank you all for doing that.”

Spencer told the children that, as her younger brother, he was the “first beneficiary of Diana’s kindness to young people.” She looked after him as a baby and at school in King’s Lynn, Norfolk. “The first day I was there, the head teacher was teaching Diana in her class and she said she was impossible, she was wriggling around and not able to pay attention,” Spencer recalled.

The teacher told Diana, ” ‘Fine, go and check on him now,’ ” said Spencer. “And she left the class and came to my classroom and came back and said to the headmistress ‘He’s fine!’ and sat down. She always had this affinity for young children.”

Before the awards, Spencer posed for pictures with some youngsters including Katie Krzyzanowski, 17 .

Katie, who has scoliosis, works as a volunteer for the Back Up charity. “It’s great that the best part of Diana is here – her own flesh and blood,” her mother Julie tells PEOPLE. “This is a legacy of Diana and to have her brother here to present is perfect.”

For Spencer, “It’s so nice for me that [the honorees] are carrying her name forward, because there’s a whole generation now for whom she’s just a historical figure rather than somebody they can actually remember,” he tells PEOPLE. “If they’re going to carry her name forward in a way that’s just so perfect, I think that’s fantastic for her and I try and tell them why it would mean so much to her as well.”

As the 20th anniversary of her passing approaches, the charity is celebrating her legacy. “Both my nephews support it, and it’s a very easy one to support,” Spencer adds. “Looking forward, of course there will be a huge build up of interest in Diana’s story as we get nearer to the 20th anniversary. That’s inevitable with these anniversaries. The 10th doesn’t seem that long ago.”

Of the possible commemorations, he says, “I don t know if it’s necessary to have a focal point. The date itself will generate its own interest. To me, as her brother, it’s just so wonderful that there are still so many positive things still stemming from her memory – that’s what I always wanted to be the case. The Diana Award to me is the absolute crux of it all because it’s taking it through the younger generation.”