The actor said he isn't leaving acting to pursue racing full-time
Credit: Charley Gallay/Getty

We’ve got good news for fans who are still heartbroken over Patrick Dempsey‘s departure from Grey’s Anatomy – the actor isn’t ready to trade in his acting career for racing full-time just yet.

“I’m not retiring from acting,” Dempsey told PEOPLE Saturday in Indianapolis, where he was in town for the annual Indianapolis 500, which Dempsey is set to kick off Sunday by serving as honorary starter, waving the green flag as a signal for drivers to start their engines.

Still, “I definitely love racing,” he said during a TAG Heuer pre-race reception. The luxury watch brand is the official timekeeper of the Indy 500 and a sponsor of Dempsey’s racing team. Dempsey, who was decked out in a charcoal gray jacket and jeans along with the same model of watch he is set to present to the Indy 500 winner (along with a $10,000 check), a TAG Heuer Limited Edition Indy 500 Chronograph, also plans to attend Indy’s Snake Pit Music Festival on Sunday.

“I love the camaraderie and the fellowship,” he said of racing. “I think the challenge is with oneself and not so much your fellow competitors, necessarily.”

He picked up his dad’s interest in racing early on, he said, and remembers going out to the family station wagon, laying on the front seat and listening to the Indy 500 broadcast when he was a boy. “It started way back then,” he explained.

Despite not giving up acting, Dempsey plans to continue racing around the world.

“I’m getting ready to go to Le Mans in about a week’s time,” he told PEOPLE. “I just love it. It’s really stimulating both physically and mentally.”

Related Video: Ellen Pompeo on Dempsey’s Exit

Other upcoming non-racing gigs also include starring in The Limit for Sundance TV and Fodors, a travelogue spy thriller.

Meanwhile, when he’s not in a race car, he’ll be tooling around in his Porsche, a prospect he actually finds rather nerve-wracking.

“I’m much more nervous driving on the freeway than I am a race car,” he says. “I don’t think there are a lot of very good drivers out there.”