"Aryn is unilaterally parenting our children without any input from me; her marginalizing of me as their father is deeply disconcerting," Williams alleges in the documents
Art Los Angeles Contemporary 2016 Opening Night
Credit: Stefanie Keenan/Getty

Jesse Williams is alleging that his estranged wife Aryn Drake-Lee is “punishing” him for his “hard work” on Grey’s Anatomy.

In his declaration filed Monday, Aug. 21 to the Superior Court of Los Angeles and obtained by PEOPLE, Williams, 36, claims that “Aryn uses the fact that I work to support our family to marginalize me as a parent.”

“I am in my 9th year on Grey’s Anatomy. The structure and demands of my work schedule are the same as they were when we married and when we decided to have our wonderful children. Working full time did not keep me from being a doting parent then and it should not now. When Grey’s is on hiatus (approximately April through late July), it is still my responsibility to earn an income, provide for our family and further my career,” the documents state.

“Despite Aryn’s refusal to recognize the time, effort and strategy required to be a self-employed professional in my line(s) of work, and her attempt to paint my work as purely recreational, all of the trips she listed in her Declaration generate the income used to support our family. I am always trying to work more efficiently so that I can free up as much time as possible to be an even more present parent to Sadie and Maceo,” the documents read.

Williams continues in the documents, “My hard work has allowed Aryn the incredible privilege of being a stay-at-home mom, with a full-time nanny, able to be with our children when she wants. She is now punishing me for providing that privilege.”

In a statement to PEOPLE, Drake-Lee’s attorney said, “Aryn continues to work tirelessly to maintain Mr. Williams’ bond with their children and it is important to remember that his allegations are one-sided. It is a shame that Mr. Williams has chosen to place the children and his family’s transition in the public domain.”

His filing comes days after Drake-Lee demanded sole custody of the couple’s two children — daughter Sadie, 3, and son Maceo, 2 — and further asked the court to restrict Williams from introducing new girlfriends to the kids before any future relationships hit the six-month mark, according to TMZ.

According to the previous documents obtained by TMZ, Drake-Lee additionally cited flare-ups of Williams’ temper, including an alleged death threat he made during a road rage incident, as another reason why she be the sole custodian of the children.

But Williams is claiming that “Aryn’s declaration is riddled with fabrications, misstatements and characterizations.”

“I am equally capable of caring for Sadie and Maceo. When I was not working on Grey’s Anatomy, and during the show’s hiatus, I fed the children, changed them, bathed them, dressed them, designed activities, played with them, read to them and put them to bed. I have always played a constant and significant role in their lives,” the Grey’s Anatomy actor states in the documents.

The documents continue, “Like Aryn, I should have sufficient time with Sadie and Maceo each week, including overnights, to continue to strengthen the very special bond I share with them and share equally in the parenting role. Aryn’s clear intent, as shown throughout her testimony is to marginalize and dismiss me as a parent, micromanage my time with the children and to exclude me from all decisions regarding our children, no matter how important.”

Williams also refutes Drake-Lee’s previous claims that he is a “bully” to her.

“I do not ‘bully’ Aryn as she claims,” the documents read. “The only aggression our children have witnessed between us occurred during the numerous times Aryn has screamed at me in their presence, including disparaging me and, after we separated, when she repeatedly slammed the front door on my leg while yelling at me in front of the children.”

“Aryn is unilaterally parenting our children without any input from me; her marginalizing of me as their father is deeply disconcerting,” he alleges.

WATCH: 5 Things You Need to Know About Actor and Activist Jesse Williams

On Wednesday, Drake-Lee responded to Williams’ previous “claim that he has a First Amendment right to publicly post images of the parties’ children” on social media.

In court documents obtained by PEOPLE, Drake-Lee is requesting “that neither parent shall publicly post otherwise private images of the parties’ children without the other parent’s consent or Court order; they may, however, share such images with family and personal friends, even over social media provided that social media account used be private and populated only by family and personal friends known to the parties.”

Williams officially split from Drake-Lee in April after five years of marriage. He addressed the split — and rumors of infidelity — in JAY-Z‘s short film Footnotes for 4:44, a visual accompaniment to the rapper’s new album, which was released in July, according to E! News.

Without mentioning his ex by name, Williams, who is dating former Friday Night Lights star Minka Kelly, subtly referenced the rumors and revealed how difficult the split was.

“I was in a relationship 13 years, 13 real years, not 5 years, not 7 years — 13 years,” he said. “All of a sudden motherf—ers are writing think-pieces that I somehow threw a 13-year relationship — like, the most painful experience I’ve had in my life with a person I’ve loved with all of my heart — that I threw a person and my family in the trash because a girl I work with is cute.”

An attorney for Williams did not respond to PEOPLE’s request for comment.