"We weren't truly getting to know each other as well as we could have," the pilot tells PEOPLE

On The Bachelor, conflict amongst the contestants is par for the course — but Peter Weber‘s season has been particularly dramatic.

First, a handful of women turned against Alayah Benavidez, accusing her of being “fake” and acting like a “different person” when the cameras were on. Weber sent her home, then changed his mind and invited her back on the show. Widespread outrage ensued, and he had no choice but to eliminate her again, this time for good.

As soon as Benavidez was out of the picture, a fresh feud emerged between Kelsey Weier and Tammy Ly, with Ly accusing Weier of “drinking excessively” after a mental breakdown. Ly also took issue with another contestant, Mykenna Dorn, and after a seemingly endless back-and-forth, Weber cut both Ly and Dorn from the competition this week.

So how does the man at the center of it all feel about the drama that dominated so many of his conversations? Well, it wasn’t ideal, to say the least.

“A lot of times it was the women bringing things up, and that would be how the conversation would start,” Weber tells PEOPLE. “That was tough for me. I didn’t really want to spend my time talking about other women with who I was with at that point. But it occurred multiple times throughout this season, and I kind of had no choice but to give in and get to the bottom of what they were concerned about.”

As a result, “We weren’t truly getting to know each other as well as we could have,” he says. “I do wish that the conversations wouldn’t have been so focused on each other.”

Credit: Maarten de Boer/ ABC

When it came to Benavidez, Weber, 28, says he “felt very pulled to her for a reason I can’t really explain.”

“I really wanted to focus on that relationship with myself and her,” he says. “When you like someone, you feel for someone. There was obviously a lot of opinions in the house about her, and unfortunately that caused a lot of drama and a lot of controversy. It ended up ultimately being a big reason of why I had to let her go, just for the greater good of it all with the house and how toxic of an environment it became with her there. That was tough.”

Credit: John Fleenor/ ABC

Weber says had Benavidez been the one, things would have been different.

“No matter how hard it would have been, I wouldn’t have let her go if I legit thought, ‘This is my wife here,’ ” he says. “But it became too much and I wasn’t that invested in her at that point. So it wasn’t worth it to take that chance and wreak all this havoc.”

The pilot admits he was “very surprised” by how quickly the issues with the girls escalated.

“There’s so much I don’t see that occurs throughout filming — I’m only there for my interactions with each of the women face-to-face,” he says. “But a lot of the conversations that I’ve seen now take place … there’s a lot of stuff that I just discovered by watching, like the girls attacking each other. I truly did not know that ever really occurred to that extent.”

He was particularly disappointed with Ly’s conflict with Weier.

“Those accusations of Tammy calling [Kelsey] out for being a pill popper and drinking too much and attacking her, I didn’t support that at all,” he says. “I wasn’t cool with it and that was just sad for me to see.”

Kelsey and Tammy
Credit: Maarten de Boer/ABC

Will you accept this rose? Sign up for PEOPLE’s free weekly Bachelor Nation newsletter to get the latest news on The Bachelor, the Bachelorette and everything in between!

Ultimately, Weber, who came in third on Hannah Brown‘s season of The Bachelorette, says while he doesn’t regret his choices, “there were mistakes that were made, and I learned from those.”

“There was a lot of wasted time on unnecessary stuff,” he says.

The Bachelor airs Mondays (8 p.m. ET) on ABC.