Racegoers and fashionistas gathered together in Berkshire Tuesday for the annual Royal Ascot, a five-day horse race attended by Queen Elizabeth and members of the royal family.

Queen Elizabeth

Matthew Childs/Reuters/Landov

The event attracts the best thoroughbreds, jockeys and trainers in the world, but it’s the fashion that really gets all the attention — especially the hats!

There are a few major dos and don’ts when it comes to hats at Ascot, and there’s a strict dress code in place. Women must not show bare midriffs or shoulders. A hat is acceptable, but no fascinators. For men, black or gray suits are acceptable, and a top hat is required.

Royal milliner Rachel Trevor-Morgan, who has been designing hats for the Queen since 2006 and is the only milliner to hold the Royal Warrant of Appointment to the Queen, tells PEOPLE: “Royal Ascot rules are very strict — you must wear a hat or headpiece that is at least 10 cm wide. Your hat will be the most memorable part of your outfit, so try to make as much effort as possible.”

“I have to make sure that my clients are clear on the size rules so that they are not turned away from the Royal Enclosure,” she adds. “There couldn’t be a worse fashion faux pas in the hat world!”

Once the perfect hat is selected, the question is how to best wear your hair.

Jane Taylor, a favorite milliner of Princess Kate and Sophie, Countess of Wessex, tells PEOPLE: “I would advise wearing your hair neatly up or away from your face, and definitely never wear sunglasses with a hat.” (Taylor is expecting Sophie to wear her designs this week.)

As for emerging trends, she says to look out for “more large, beautiful brims — the bigger the better! They are so flattering and really frame the face.”

“Lines are quite clean and dramatic offset with softer florals and feathers,” Trevor-Morgan adds. “The color palette has been quite soft tones with highlights of brights.”

Queen Elizabeth and Prince Harry

Samir Hussein/WireImage

And all eyes were on the Queen as she made her royal entrance in a horse-drawn carriage alongside Prince Philip and Prince Harry.

Harry, who spent the morning meeting First Lady Michelle Obama at Kensington Palace, looked royally dashing in his black suit and top hat.

And the Queen stunned in a floral dress by Peter Enrione paired with a fuchsia coat and matching hat, which was designed by her personal dressmaker, Angela Kelly.

As Trevor-Morgan says, “She has probably worn more hats than anyone else in the world. She always looks so beautifully finished. When people think of the Queen they often think of her hats. She is one of the few people who can wear such an array of colors and always look so elegant.”

It’s not just royals who are getting in on the hat action. Socialites, celebrities, royal fans and racegoers alike don their best headwear for the fashion-forward social event of the season.

“Women are generally wearing less hats day-to-day, so I think Ascot has become hugely important as the main hat wearing event of the year,” Trevor-Morgan says. “Women love hats and there simply aren’t enough hat-wearing occasions, and so they are embracing Royal Ascot more and more.”

What do you think of the strict dress code at the Royal Ascot? How about the Queen’s bright hat? Sound off below.

–Erin Hill, with reporting by Monique Jessen

RELATED VIDEO: Queen Elizabeth, Prince Harry and Prince Charles Attend the Opening of the Chelsea Flower Show