As Stephen King's It hits theaters, a look at the roots of clown phobia
It_03192017_Day 61_18998.dng
Credit: Warner Bros. Entertainment Inc.

Why are so many people afraid of clowns?

With a big-screen version of Stephen King‘s It, featuring bloodthirsty supernatural clown Pennywise, hitting theaters Sept. 8, and the next season of American Horror Story hinting at a cornucopia of creepy clowns when it returns Sept. 5, those with coulrophobia are in for a very unsettling fall.

But horror movies aren’t to blame for many people’s anxiety about clowns. Steve Scholzman, M.D., an assistant professor of psychiatry at Harvard Medical School, tells PEOPLE the fear is much more deep-seated, relating to one of our most primal instincts — pattern recognition.

“Pattern recognition allows to see when things are recognizable but off just enough to where you take caution,” Scholzman explains. “That’s why a little kid can recognize something as being a little bit dangerous without quite knowing why they know it’s dangerous. They recognize it as familiar but not quite the same as what they’re seeing.”

It_08312016_Day 46_11374.dng
Credit: Brooke Palmer/Warner Bros.

Bill Skarsgard as Pennywise in It.

An example of this is how we can recognize that a pug and a German shepherd look very different but are both still dogs. Clowns play with this instinct when they distort what the human face should look like.

“With clowns, you have two arms and legs and a face — it’s human,” Scholzman says. “Like most monsters, it’s a recognizable shape that’s tweaked. The perpetual smile is by itself is a different type of pattern recognition because we look at people’s expressions to get a sense of what to feel upon seeing them. But a clown doesn’t do anything but smile, so you don’t really know what it’s thinking or feeling. And all of those things are playing on pattern recognition.”

Scholzman says that clown phobia isn’t actually as prevalent as pop culture makes it out to be. Real cases of the phobia, in which someone wouldn’t be able to be in the same room as a clown, are “pretty rare,” according to Scholzman.

Warning: there’s a scary clown in the trailer above. Proceed at your own risk.

Yet people continue to be unnerved by clowns, which has been leading to real-life professional clowns losing jobs over the phobia, according to The Hollywood Reporter. And some of them are blaming It for perpetuating the fear.

“It all started with the original It,” World Clown Association president Pam Moody told THR. The 1986 novel and 1990 miniseries “introduced the concept of this character. It’s a science-fiction character. It’s not a clown and has nothing to do with pro clowning.”

The WCA is preparing for the upcoming American Horror Story: Cult and It onslaught this time around, sending out a guide to its membership about the differences between the scary pop-culture clowns and real-life entertainers.

Last year, a global fad led to a spate of people dressing up as scary clowns and frightening or threatening locals. A handful of people were arrested — and the hysteria dealt yet another setback to the clown community.

“People had school shows and library shows that were canceled,” Moody said. “That’s very unfortunate. The very public we’re trying to deliver positive and important messages to aren’t getting them.”