A source tells PEOPLE that the actor, who has been staying out of sight since leaving the home he shares with Stewart in Los Angeles, recently asked her to remove her belongings from the house.
And early Saturday morning, a U-Haul truck was spotted in the driveway.
News of Stewart’s infidelity – she cheated on Pattinson with her Snow White and the Huntsman director Rupert Sanders, who is married with kids – first broke last week. The 22-year-old actress subsequently apologized to Pattinson for cheating, and sources told PEOPLE they hadn’t spoken since until now.
“The entire thing has been such a blow to Rob. Talk about having his heart ripped out,” a source who knows the couple tells PEOPLE. “Rob just isn’t going to be able to get past the cheating – one time or not.”
Nevertheless, the two are scheduled as co-presenters at MTV’s Video Music Awards on Sept. 6.