"I'm jealous of people who can put on sweatpants and go get a coffee," the star tells PEOPLE

Evan Agostini/PictureGroup

Zooey Deschanel‘s name might conjure images of her “adorkable” New Girl character, Jess, but in daily life, the actress doesn’t feel tied to a specific, signature look.

“I’m completely [by the] seat of my pants,” she tells PEOPLE of choosing outfits. “I’m totally an intuitive decision maker — I am never calculating, especially about that sort of thing. I’ve always just been interested in wearing what I like.”

Though Deschanel isn’t opposed to experimenting with her look — which she simply describes as “feminine and fun” — there is one item you’ll never see her wearing: sweats.

“It’s so annoying,” she says, “every time I leave my house I’m supposed to look polished … I’m jealous of people who can put on sweatpants and go get a coffee.”

It’s the same story with footwear. “Sometimes I don’t realize how uncomfortable [my shoes are] until I’m out of the house,” she shares. “There have been a number of times at events where I’ve been like, ‘I wish I was wearing flats.'” Zooey, we feel your pain. Tell us: What’s the one item you’d never be caught dead wearing?

–Jennifer Cress, reporting by Jackie Fields