Sebastian Stan and Tom Hiddleston may play foes in the Marvel Cinematic Universe, but offscreen, Stan wanted to reach out and help Hiddleston during the actor's whirlwind romance with Taylor Swift

Sebastian Stan and Tom Hiddleston may play foes in the Marvel Cinematic Universe, but offscreen, Stan wanted to reach out and help Hiddleston during the actor’s whirlwind romance with Taylor Swift.

During an appearance on Thursday’s Watch What Happens Live with Andy Cohen, Stan, 35, was asked by a caller about Hiddleston and Swift’s highly-publicized pairing in 2016 — and that iconic “I Heart T.S.” tank top the English actor wore at her Fourth of July party.

“It was one of awe and extreme concern,” Stan said about his reaction to seeing Hiddleston in the shirt. “Obsession and yet at the same time, I don’t know, I was really worried — mainly for him. Love is quick and hard.”

Asked if he’s talked to Hiddleston about it, Stan said, “No, no I didn’t. I actually forgot about that incident!”

Want to keep up on the latest from PEOPLE? Sign up for our daily newsletter to get our best stories of the day delivered straight to your inbox.

Hiddleston and Swift dated for three months in the summer of 2016 before going their separate ways.

Things were hot and heavy during their short courtship. The pair met each other’s parents, celebrated July 4th together at Swift’s annual star-studded bash and traveled around the world, enjoying PDA-filled trips to Italy, Australia and Great Britain.

*PREMIUM EXCLUSIVE* Taylor Swift and Tom Hiddleston have a dinner date **WEB EMBARGO UNTIL 10AM PST ON 07/29/16**
Credit: AKM-GSI

While it may have been short-lived, Hiddleston told GQ in their March 2017 issue that it was definitely “real.”

RELATED VIDEO: Who is Most Likely to Ruin a Take on the Set of ‘Captain America: Civil War’?

Taylor is an amazing woman. She’s generous and kind and lovely, and we had the best time,” he said, going on to suggest that the pressures of dating in the public eye contributed to their split. “A relationship in the limelight… A relationship always takes work. And it’s not just the limelight. It’s everything else.”

You have to fight for love,” he added. “You can’t live in fear of what people might say. You know, you have to be true to yourself. … I have to be so psychologically strong about not letting other people’s interpretations about my life affect my life. A relationship exists between two people. We will always know what it was.”

EXCLUSIVE: Taylor Swift celebrates the 4th of July with boyfriend Tom Hiddleston, Cara Delevinge, Karlie Kloss, Ruby Rose, Blake Lively, Ryan Reynolds, and Gigi Hadid at Taylor's house in Rhode Island
Credit: Ryan Turgeon / Splash News

As for that “I Heart T.S.” tank, Hiddleston previously defended that, too.

“The truth is, it was the Fourth of July and a public holiday and we were playing a game and I slipped and hurt my back,” he said. “And I wanted to protect the graze from the sun and said, ‘Does anyone have a t-shirt?’ And one of her friends said, ‘I’ve got this’ … And we all laughed about it.”

“It was a joke. Among friends,” he continued.

Watch What Happens Live with Andy Cohen airs Sunday through Thursday (11 p.m. ET) on Bravo.