The former Broncos quarterback joked that Tom Brady isn't allowed to leave Florida after he accidentally walked into the wrong house last month

payton manning, tom brady
Peyton Manning, Tom Brady
| Credit: Michael Loccisano/Getty Images; Lorenzo Bevilaqua/getty images

Tom Brady, Phil Mikelson, Tiger Woods and Peyton Manning participated in a video conference Thursday ahead of their upcoming golf match — and the former Broncos star wasted no time roasting his fellow quarterback.

"Of course, you know, the tournament had to be in Florida — you know, after Tom's B and E arrest," Manning, 44, joked, referring to Brady's recent mishap, when he walked into the wrong house while trying to visit Buccaneers' offensive coordinator Byron Leftwich's home. He was not actually arrested for breaking and entering.

"With the ankle monitor, he couldn't leave the state, so it had to be in Florida," Manning continued, going on to joke that he and Woods spoke to the sheriff in Tampa, who cleared Brady to travel to Palm Beach for the charity event on May 24.

Manning added that he wished the tournament were being held "in a place where they don't like Tom very much."

"Indianapolis, Denver, Boston — you know, after he just betrayed them and broke their hearts," he said, prompting the new Buccaneers player to laugh and cover his face in his hands. After 20 seasons with the New England team, Brady announced earlier this year that he was heading to Tampa Bay. "So Palm Beach is the best we can probably do."

The upcoming two-on-two golf match, which has been dubbed The Match: Champions for Charity, will see Woods and Manning team up to face off against Mickelson and Brady.

"Look, I think the teams are fair," Manning continued during the Thursday call. "Phil chose the right partner in Tom. Together, you know, they have 11 championships."

The two-time Super Bowl champ then added, "Tiger and I have 17."

Golf is one of the few sports that has been allowed to continue in certain areas of the country during the novel coronavirus (COVID-19) pandemic, and the upcoming match-up will raise money for COVID-19 relief.

Manning isn't the only one who has shared a bit of playful smack talk ahead of the match.

Brady previously shared a photoshopped image of himself and Mickelson on a golf green with Manning and Woods standing next to them as children.

"Never had much of a hard time beating the colts or a tiger, don’t see this time being much different..." Brady captioned the photo.

"Nor do I 😏," Mickelson wrote in the comments.

As information about the coronavirus pandemic rapidly changes, PEOPLE is committed to providing the most recent data in our coverage. Some of the information in this story may have changed after publication. For the latest on COVID-19, readers are encouraged to use online resources from CDC, WHO, and local public health departments. PEOPLE has partnered with GoFundMe to raise money for the COVID-19 Relief Fund, a fundraiser to support everything from frontline responders to families in need, as well as organizations helping communities. For more information or to donate, click here.