"I don't think about it too much," the former tennis pro tells PEOPLE of his time spent playing the sport

During the 2012 U.S. Open, Andy Roddick announced that he’d be retiring from tennis after his run in the tournament ended. Now, seven years later, Roddick will return to the open as a partner of IHG Hotels & Resorts, and being back in New York City for the event allows him to think back on when he used to compete.

“There’s so many memories and relationships throughout the years at this tournament,” Roddick, 36, told PEOPLE at Wednesday night’s the IHG Legends, Unmatched U.S. Open celebration in New York City. “I love coming back to New York. It always feels comfortable. You see a lot of the familiar faces, so it’s nice to have this one thing in your life that you feel like you can always go back to and it gives you a hug a little bit.”

The former top-ranked tennis player in the world admits that he doesn’t usually reflect on his storied career, though.

“I don’t think about it too much,” he said. “I think that’s for other people. It seems so far away. The game has given me everything I’ve ever had in life, so I certainly appreciate it, but I don’t do much in the game of tennis now. Unfortunately, you have to; You lose control of geography if that happens a little bit.”

Andy Roddick
Credit: Dia Dipasupil/Getty
Andy Roddick
Andy Roddick competing in the IHG Hotels and Resorts Legends, Unmatched table tennis event at N.Y.C.’s Kimpton Hotel Eventi on Aug. 21.
| Credit: Dia Dipasupil/Getty

As for whether he has any career regrets, Roddick said, “Everyone has those, but that’s not my takeaway.”

The father of two admits that not being a full-time athlete anymore comes with some perks. “It’s nice to be able to have a little bit more wine at dinner without worrying about how it affects [me] the next day,” he said.

RELATED VIDEO: Brooklyn Decker Tweets How Her Husband Saved The Day With Tequila

Though Roddick has actually taken up another sport. “I play golf a lot,” the 2003 U.S. Open men’s singles winner said.

Also in his free time, Roddick enjoys traveling with his wife, actress Brooklyn Decker, and their son Hank, 4, and daughter Stevie, who will turn 2 in November.

“We have a little place in a mountain town in North Carolina that has one stoplight and dodgy cell phone service,” Roddick said. “We’re in the car and we can put on a movie for the kids. That’s kind of a bit of a getaway for us.”

Andy Roddick Foundation Gala, Austin, USA - 01 Nov 2018
Andy Roddick and Brooklyn Decker in Austin in 2018.
| Credit: Suzanne Cordeiro/REX/Shutterstock

Roddick and Decker, 32, took advantage of his IHG partnership for a kid-free trip to New York City to celebrate their 10-year wedding anniversary earlier this year.

“We got here in the afternoon and we got the room, and it was just silent. It was deafening,” Roddick quipped. “We went to a restaurant that we’d been going to for 15 years and just had a quiet night. It was 24 hours and it was great.”