Sean Penn Recounts 'Startling' Experience of Crossing Ukraine-Poland Border: 'There Was an Immediacy'

"It was as though they wanted to believe they're going to be able to come back," Sean Penn said of the Ukrainians he saw fleeing their country amid the ongoing Russian invasion

Photo: Rodin Eckenroth/Getty

Sean Penn is back home after being in the thick of the Russian invasion of Ukraine.

The Academy Award winner, 61, recalled the "startling" experience of crossing the border from Ukraine to Poland earlier this week as he appeared Friday on Anderson Cooper 360°.

"We had the luxury of being able to abandon a rented vehicle on the side of the road. This was a startling thing to me; it was mostly women and children, some in groups and some just a mother and her child, in almost all of those cars," he explained. "In some cases, the father was dropping them off and returning, because we know that from 18 to 60, men are not to leave, they're to stay in the resistance against Russia."

"I didn't see any luggage. It was as though they wanted to believe they're going to be able to come back, and there was an immediacy to leave because of the incredible amount of people leaving and how long it takes to get out of the country now. So, the car is pretty much, aside from those who have family or friends that could help on the other side, all they have," Penn continued. "So, in the several miles that we walked after abandoning our car, I didn't see one of those cars move a car length because that line was so slow. And then you get there and you see all those who have walked as well in that crowd."

"I was glad, not so much in the moment, but I was glad to have had the experience of having to see what it was to get through that border ... what it is to just sit there for sometimes days," Penn shared.

Upon his return stateside, Penn was photographed shopping at the Desert Hills Premium Outlets outside Palm Springs, Calif., on Saturday.

The actor arrived in Kyiv, the capital of Ukraine, last week to work on a documentary about the ongoing Russian invasion. He was previously seen meeting with Ukrainian troops on the frontline for the film in November, wearing a helmet and protective gear.

"I think that they have noticed that the calvary is not coming. Even this incredible refugee crisis is full of people who, but for their children, would not be leaving," Penn said on AC360.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

"And I think about — there are great Russian people who don't want this war, who don't believe in it — and getting information that supports them because obviously they're under great threat to speak out and to protest, yet some are doing it," he added. "There's such a closeness between Ukrainian people and Russian people. This thing only makes sense as an exploitation of the powerful."

Russia's attack on Ukraine continues after their forces launched a large-scale invasion on Feb. 24 — the first major land conflict in Europe in decades.

Details of the fighting change by the day, but hundreds of civilians have already been reported dead or wounded, including children. More than a million Ukrainians have also fled, the United Nations says.

"You don't know where to go, where to run, who you have to call. This is just panic," Liliya Marynchak, a 45-year-old teacher in Ivano-Frankivsk, Ukraine, told PEOPLE of the moment her city was bombed.

Chelsea White

The invasion, ordered by Russian President Vladimir Putin, has drawn condemnation around the world and increasingly severe economic sanctions against Russia.

With NATO forces massing in the region around Ukraine, various countries have also pledged aid or military support to the resistance. Ukrainian President Volodymyr Zelenskyy called for peace talks — so far unsuccessful — while urging his country to fight back.

RELATED VIDEO: Maks Chmerkovskiy Says He Feels 'Guilt' for Leaving Ukraine amid Russian Invasion: 'I'm Emotional'

Putin insists Ukraine has historic ties to Russia and that he is acting in the best security interests of his country. Zelenskyy has vowed not to bend.

"Nobody is going to break us, we're strong, we're Ukrainians," he told the European Union in a speech in the early days of the fighting, adding, "Life will win over death. And light will win over darkness."

The Russian attack on Ukraine is an evolving story, with information changing quickly. Follow PEOPLE's complete coverage of the war here, including stories from citizens on the ground and ways to help.