Tom Ford's Painful Memories of Childhood Bullying: 'I Was Absolutely Tortured'

During a recent appearance on The Jess Cagle Interview, the 55-year-old fashion designer and film producer spoke about growing up in Texas

For more from Tom Ford, watch the full episode of The Jess Cagle Interview, available now on the new People/Entertainment Weekly Network (PEN). Go to PEOPLE.com/PEN, or download the PEN app on Apple TV, Roku Players, Amazon Fire TV, Xumo, Chromecast, iOS and Android devices.

Tom Ford may have a Jay Z song named after him now, but he wasn’t always so popular.

During an appearance on The Jess Cagle Interview, the 55-year-old fashion mogul and filmmaker spoke about growing up in Texas and dealing with bullying as a child.

“As a kid in the ’60s, I wasn’t great at football, I was not great at team sports, I wasn’t great with my BB gun, which in Texas soon turns into a .410 shotgun and then turns into something else,” he tells PEOPLE and Entertainment Weekly‘s editorial director. “I wasn’t interested in those things and so I was teased a lot.”

Instead, Ford had already developed an interest in fashion, which didn’t sit well with his peers. “As a kid I had a very, I think you’re born the way you’re going to be and I came out dressed in a little blazer and little loafers,” he recalls. “When I was in the second grade I didn’t want to carry a book bag, I carried an attaché case. Of course I was tortured. I was absolutely tortured.”

When Ford headed to college in New York City, he found out what made him feel so ostracized — he realized he was gay.

Gregg DeGuire/WireImage

Despite all of his success, those painful childhood memories are still with him to this day. Ford says it took him a long time to deal with the suffering, and he still flashes back to those vulnerable times.

“I think I shoved them to the back of my mind and didn’t really confront them. I think that they’re very present to date,” he says of the memories. “Still to this day if I walk past a group of kids, they can be 8 years old playing soccer, and that ball comes towards me, I panic because, ‘My God, I have to kick that ball and they’re all going to laugh because I’m not great at soccer.’ It’s a sort of instant panic.”

Still, Ford says he had a happy childhood. At home, his parents accepted him — and continue to support him — for who he is.

When asked what Ford would tell his 7-year-old self, the advice was something he tells himself to this day.

” ‘Relax,’ but that’s what I would whisper in my own ear now,” he says. ” ‘Relax, it’s going to be okay. Relax.’ “