Carole Radziwill announced her decision to leave The Real Housewives of New York City on Wednesday

Carole Radziwill‘s decision to leave The Real Housewives of New York City was a choice to follow her heart and not her bank account, she implied on Friday while suggesting that the hit Bravo series was forcing her to go against her “value system.”

But a source close to the situation tells PEOPLE that Radziwill shouldn’t make the issue bigger than it is.

“Why not leave gracefully?” the insider says of Radziwill’s recent Instagram posts alluding to her departure. “To post a poll about being forced to do something against your morals and value system, it’s a burned-bridge type of thing to do. Why be so negative? This does not make sense.”

“She is digging in deeper and deeper — and this is all fiction now,” the source continued, suggesting there were other reasons behind Radziwill’s departure. “Everybody needs to move on. Stop dredging it up. Exit gracefully.”

A rep for Bravo did not immediately respond to PEOPLE’s request for comment.

Carole Radziwill
| Credit: Robin Marchant/Getty

PEOPLE was first to report Radziwill’s exit from RHONY on Wednesday. She said in a statement that after six seasons on the show, “I have decided to return to what I do best — journalism and producing.”

“I am sure this does not come as a surprise to any of the viewers, all of whom have been supportive, encouraging, and kind,” she added. “My original curiosity about reality television has waned over the years and I am focusing on TV and writing projects that better suit my more steady temperament.”

She ended her statement with sincerity and wit. “I have worked with amazing producers, made great friends, and I’m thrilled to leave frenemies behind,” Radziwill said. “I will remember this entire experience with delight, humor, and a veracious accuracy. Next.”

The Real Housewives of New York City - Season 9
The Real Housewives of New York City
| Credit: Patrick Ecclesine/Bravo

On Friday, the reality star polled her followers on her Instagram Story to see whether they would follow their hearts or put their bank accounts first.

“If you were doing something that was forcing you to do things that went against your own value system would you walk away regardless of the pay?” she asked. “Its [sic] a tough decision.”

After fans overwhelmingly voted “yes” on her survey, the best-selling author wrote, “So 80% would walk away from something that went against their own values. That’s how I felt this past year too. It’s difficult but life is long and you can’t always play the short game.”

It appears she is very happy with that decision. Taken while she was vacationing in Aspen, Colorado, the picture shows Radziwill with a big grin on her face. “My advice: Save for a rainy day because that day might come when the sun is shining and you can walk away with a smile,” she wrote.

Credit: Carole Radziwill/ Instagram
Credit: Carole Radziwill/Instagram

Radziwill’s run on RHONY began in season 5 amid a major shakeup. Original cast members Bethenny Frankel, Jill Zarin and Alex McCord had departed, as had Kelly Killoren Bensimon.

Frankel returned in season 7 and forged a quick friendship with Radziwill. In RHONY‘s tenth season, airing now, the two have had a pretty epic fallout, though Radziwill has remained on good terms with Tinsley Mortimer, Dorinda Medley, Sonja Morgan, Luann de Lesseps and Ramona Singer.

“I love Carole inside and out,” Singer said on Watch What Happens Live with Andy Cohen Wednesday, adding that she was “sad” to see Radziwill go. “I think she’s really warm, special, down to earth, truthful, giving, kind, supportive of women, genuine, smart [and] fabulous. I’m like, ‘Wow.’ I’m going to miss not being with her doing the next season. It’s going to be a void. This is going to be a void for me.”

RELATED: The Real Housewives of New York City Go Makeup-Free For PEOPLE’s Beautiful Issue

While Radziwill still hasn’t gotten specific about what exactly was going against her “value system,” PEOPLE learned her fight with Frankel influenced the choice.

“They never seemed to be able to reconcile,” a source previously told PEOPLE. “The show couldn’t go on another season with both of them at each other’s throats. And Bethenny’s an O.G. She’s been there since the beginning. She’s a [piece of work], but she gives the show a lot of energy. There’s no way they’re going to do it without her. Last time she left, the show went to hell.”

However, one production source insists that as far as producers are concerned, Radziwill could have chosen to stay: “She was not pushed to leave. She left on her own accord.”

The Real Housewives of New York City airs Wednesdays (9 p.m. ET) on Bravo.