"Of course" he and Angelina Jolie will add to their brood, Pitt tells PEOPLE

Brad Pitt says three isn’t enough. At the Palm Springs International Film Festival on Saturday, the actor told PEOPLE that he and Angelina Jolie want another sibling for Maddox, 5, Zahara, 2, and 7-month-old Shiloh. “Always, of course,” he said. “We’ll let you know when we get there.”

Pitt also said that the globe-trotting clan (who recently traveled to Panama) will add to their frequent-flier miles, temporarily relocating to New Orleans for the next several months to film The Curious Case of Benjamin Button.

“I do like it there,” Pitt said. “It’s one of our most unique cities and it certainly needs some help right now. I know it will do fine, and the people are just fantastic.”

Pitt has visited New Orleans for the Hurricane Katrina recovery effort, cosponsoring with Global Green USA a contest to find an eco-friendly plan for housing and community centers. He also has been working with organizations helping AIDS orphans.

The family’s most recent stop has been the Caribbean, where Pitt and Babel costar Cate Blanchett were on location for Benjamin Button.

“We wrapped at 6 in the morning, got on a plane and got our crusty asses here,” he told PEOPLE.

Jolie and the children stopped over in Los Angeles while Pitt headed to Palm Springs with Blanchett for the film festival’s awards gala. The festival honored the Babel cast as well as Blanchett for career achievement and director Alejandro Gonzélez Iñérritu. (The movie is also nominated for several Golden Globes, including a best supporting actor nod for Pitt.)

“For us, it’s about the ensemble award,” said Pitt. “Given the international nature of our cast, it’s a really nice honor.”

During the gala, Pitt joined the cast on stage and the 1,800 people in the audience in singing happy birthday to costar Rinko Kikuchi, who turned 26 on Saturday.