Jenna Fischer says some fans "came down real hard" on Pam Beesly for briefly moving away from Jim Halpert to attend art school

THE OFFICE -- "Couples Discount" Episode 916 -- Pictured: Jenna Fischer as Pam Halpert
Jenna Fischer as Pam Beesly
| Credit: Byron Cohen/NBCU Photo Bank/NBCUniversal via Getty

Jenna Fischer is reflecting on some of the criticism that was directed at Pam Beesly on The Office.

During the latest episode of the Office Ladies podcast, Fischer (Pam) spoke with co-host Angela Kinsey (Angela Martin) about how some fans "were not happy" with Pam's decision to briefly move to New York City to attend art school. 

While breaking down season 5 episode "Weight Loss, Part 2," Kinsey, 49, noted that people were "fired up" that while Pam was seen laughing with new friends, her then-boyfriend Jim Halpert was talking about how much he missed her.

"People were not happy with Pam when this episode aired, I remember," said Kinsey.

"I just want to say that when Pam went to art school for only three months when she was not yet married and didn't have children, people came down real hard on her because of Jim's feelings," said Fischer, 47.

Fischer went on to note that later on in season 9, when Jim makes some big moves of his own to chase his dreams, some fans again criticized Pam. 

"Then, when Jim invested in a business without telling Pam, and left his family, and left Pam alone to care for their two children while he chased his dreams, people came down on Pam again for not being supportive enough of Jim," she said. "So in both of these instances I just want to say I noticed a lot of hate for Pam. Do I want to call it sexist? I do." 

"Pam faced a lot of scrutiny as a character I think because she was a woman, right?" she asked Kinsey.  

"That river runs deep," her costar added, as Fischer replied, "It runs deep and it's going to run long into some future seasons." 

Want to get the biggest stories from PEOPLE every weekday? Subscribe to our new podcast, PEOPLE Every Day, to get the essential celebrity, entertainment and human interest news stories Monday through Friday.

The Office
John Krasinski and Jenna Fischer
| Credit: hris Haston/NBCU Photo Bank/NBCUniversal via Getty

Fischer has opened up in the past about the lasting impact of Jim and Pam's romance on fans — and how some viewers have a hard time accepting the fact that she and John Krasinski aren't together in real life.

"People don't know how John and I are not a couple in real life," Fischer previously told Brian Baumgartner, who played Kevin Malone on the series, on An Oral History of The Office.  

"I don't know how to explain it, because it's a little bit like telling kids there's no Santa," she added. "The bottom line is that we were playing characters, but I know that if people think of us as John and Jenna, then it's destroying some of the magic of Jim and Pam." 

In real life, Krasinski has been married to actress Emily Blunt since 2010, and Fischer wed writer and director Lee Kirk the same year.