"In the second series, I didn't think it was fair to Prince Philip, to the Duke of Edinburgh, based on very little," says Julian Fellowes

Julian Fellowes is coming to Prince Philip‘s defense.

The creator and writer of Downtown Abbey — the Golden Globe and Emmy-winning PBS drama series that chronicled the aristocratic Crawleys in the early 20th Century — is speaking out about Prince Philip’s portrayal in the second season of Netflix’s The Crown — and why he believes it wasn’t “fair” to the 97-year-old Duke of Edinburgh.

Matt Smith in The Crown (L) and Prince Philip (R)
| Credit: Robert Viglasky/Netflix; Hulton Archive/Getty

The Crown is a wonderful piece of work, and a brilliant, brilliant writing from Peter Morgan,” Fellowes, 68, tells Katie Couric in the new episode of her podcast, the latest installment of her editorial initiative with the BBC, which has also featured conversations with Sharon Horgan and Graham Norton.

“It was beautifully acted, beautifully written,” he says. “For me, I’m not completely comfortable with dramatizing people who are still alive and still living their lives. Because I think it’s possible to be unfair. And in the second series, I didn’t think it was fair to Prince Philip, to the Duke of Edinburgh, based on very little.”

Season 2 of The Crown was released Dec. 8 on Netflix and stars Claire Foy (Queen Elizabeth) and Matt Smith (Prince Philip).

Fellowes explains, “Now I’ll be punished for that because it’s a great success and it deserves to be. I don’t know. I think when people are still alive, living their lives, doing a good job and popular and loved, do they deserve it? And in that sense, I’m not sure they do.”

Couric then inquired if Fellowes believes “that a lot of artistic license was taken in the storylines in terms of how these individuals have been portrayed in the series?”

“You’re getting me into a tricky area here,” says Fellowes. “I think that a lot of it was based on obviously very good research, but some of it was not. Some of it was extrapolation from a rumor or someone’s rather prejudiced account. And then it was presented as fact. I’m not sure that’s just.”

RELATED VIDEO: Claire Foy Responds to The Crown Pay Controversy

Although the Downton Abbey creator doesn’t think the storyline was completely accurate, he applauds The Crown creator Peter Morgan.

“But I’m a big fan of Peter Morgan, I repeat that. I think he’s the best writer on television at the moment,” says Fellowes. “And it’s deservedly successful as far as I’m concerned.”

  • Watch Elizabeth and Philip: A Royal Romance now on the new PeopleTV network. Go to PeopleTV.com, or download the PeopleTV app on your favorite mobile or connected TV device.

The second season of the Netflix show was set in the early 1960s, and much of the drama surrounded Queen Elizabeth sending husband Prince Philip off on a five-month tour, which is among the stresses explored.

“When they finally get a chance to look at their marriage, it’s in pretty dire straits,” Foy told PEOPLE. “They’ve both in their own ways dealt with it and made themselves feel better or healed themselves in whatever way they are able to.”

Part of their “healing” involved welcoming their two younger sons, Prince Andrew (born in 1960) and Prince Edward (1964).

The season was also a departure from the “fairy story” of last year’s, said director Philip Martin. By the early 1960s, the public “is questioning whether we need a royal family,” he said. “It’s a complete energy change.”

There are also flashbacks to Philip’s lonely childhood. (His own royal family was ousted during the coup in Greece, and his mother was institutionalized.)

“We learn a lot more about him and his history and his heritage,” said Smith, 35, who portrayed him in the series. “It has made him quite tough and resilient.”