Jon Gosselin Doesn't Have Contact with Most of His Kids: They 'Aren't Talking to Me'

"If they don't want to talk to me, that's fine, as long as they keep up with their sibling connections," Gosselin says

By

and

Published on December 12, 2018 10:23AM EST

Jon Gosselin says he has been awarded temporary sole custody of his teenage son Collin.

For over two years now, Collin has been living away from his family in a program for children with special needs. Jon’s ex-wife Kate Gosselin previously told PEOPLE that she made the difficult decision to enroll her son in the program in order to help him reach his full potential.

When Collin leaves the program later this year, he will go home to live with Jon.

“It was just easier for the transition,” he told reporters at WE tv’s Real Love: Relationship Reality TV’s Past, Present & Future panel on Tuesday in Los Angeles. “So it’d be easier for one parent to just transition him home, and he wanted to live with me, so it was easier to do it that way.”

“I’m excited to have a male around,” he added. “I’m surrounded by females.”

PEOPLE confirmed in November that Jon and his attorney submitted papers to a Pennsylvania court on Oct. 24 requesting that Collin live with him, a decision he claims would be in Collin’s “best interest.” In November, Kate, and her lawyer reportedly did not show up to court after her request to postpone the hearing was denied.

Reps for Jon and Kate have not responded to PEOPLE’s request for comment.

Collin is one of Jon and Kate’s 14-year-old sextuplets — along with Alexis, Leah, Hannah, Joel and Aaden. (They also share 18-year-old twins Mady and Cara.)

Jon also said Hannah lives with him “full time.”

“Hannah lives with me full time, and then Collin,” he said. “Leah, Alexis, Joel and Aaden, they’re with their mom, and then the twins are 18, so they’re adults.”

And while Jon said his relationship with Collin and Hannah is strong, he doesn’t have contact with the other children.

“The other four aren’t talking to me, but they’re talking to Hannah,” he said. “It’s what the kids want to do. If they don’t want to talk to me, that’s fine, as long as they keep up with their sibling connections.”

The former reality star recalled that last time he saw all of his children, admitting that the meeting was tense.

“It was just volatile and a lot going on,” he said. “It was just not a good time.”

He added that he does not “communicate” with Kate either. “It’s too much right now,” he said.

kate-gosselin — Kate Gosselin/Instagram; Dimitrios Kambouris/Getty

Jon, 41, has documented his time with Hannah this year — including on the Fourth of July, on Father’s Day weekend, at a restaurant in May and on her birthday that same month. His photo of Hannah on her first day of school back in August seemingly suggested that she was attending a different school than her siblings, who all posed for a different back-to-school photo shared to social media by their mother.

In May, Jon and Hannah visited Collin and shared birthday cake and pizza together. Later that month, Hannah posted a photo collage with Collin and their father.

In November, Jon and Hannah reunited with Collin again, posing together in front of a door.

In 2016, twins Mady and Cara told PEOPLE that they do not talk to their father. “I wouldn’t even know what to say about him,” Cara explained.

”He makes it seem like we’re being kept from him, which is insane,” Mady said. “He should maybe spend some time thinking about why we don’t want to see him, and maybe realize that if he ever does want a relationship with us, talking about us on TV is not the way to make that happen.”

Reporting by RAHA LEWIS