By Jodi Guglielmi and Phil Boucher
October 04, 2016 10:30 AM

In the wake of their shocking split, Brad Pitt and Angelina Jolie fight over the future of their six kids. Subscribe now for exclusive new details on the turmoil they hid for years, only in PEOPLE.

Filming By the Sea wasn’t quite the “unconventional honeymoon” Brad Pitt and Angelina Jolie hoped for.

Just after they wed in 2014, Jolie, 41, and Pitt, 52, jetted off to the Mediterranean nation of Malta with their six children to begin filming the ’70s drama about a troubled married couple, directed and written by Jolie.

In real life, the pair’s relationship already appeared to be rocky. Those on the set of the movie tell PEOPLE that Jolie and Pitt seemed distant and unhappy.

“They hardly spoke when they were together,” a set source tells PEOPLE in this week’s cover story, adding that they often took separate business trips on the weekends.

Brad Pitt Angelina Jolie People Cover

Watch more on People Cover Story: Brad & Angelina’s Family Secrets, available now on the new People/Entertainment Weekly Network (PEN). Go to, or download the PEN app on Apple TV, Roku Players, Amazon Fire TV, Xumo, Chromecast, iOS and Android devices.

Jolie and Pitt rented three villas for themselves, the kids and eight nannies, the source says. Due to their celebrity status, the family lived mostly in isolation while filming. The kids played in the pool, but “they couldn’t go anywhere on the island because of who they are,” says the source. Their oldest son, Maddox, 15, pitched in as a production assistant and was “a really nice guy,” the source says.

When they weren’t filming, Jolie and Pitt “were quite distant” with each other, adds the source.

During his off-time, a downbeat Pitt would go to a local bar for a light beer, never more than two.

“He would be very quiet and just keep to himself,” says an onlooker. “Angelina was never with him.”

Says the set source: “It was not a honeymoon.”