For the outing, Baldwin, 21, sported denim shorts paired with a leather jacket while Bieber, 24, wore a blue KTH X Coke sweatshirt with green pants.
Neither was, however, wearing a wedding ring.
The sighting comes just one day after the lovebirds tied the knot at a New York City courthouse.
“They went ahead and did it without listening to anyone,” a source close to the couple tells PEOPLE.
Although they’re already Mr. and Mrs. Bieber, the couple still plans on having a formal ceremony. “They’re going to have a big blowout, in front of God and everyone they love,” a religious source who spoke to the family confirmed to PEOPLE.
Another source tells PEOPLE their wedding remains “their focus.”
“They are insanely happy and can’t wait to celebrate their love with friends and family,” the source says.
TMZ also reported that the couple appeared to be extremely emotional with Bieber telling Baldwin, “I can’t wait to marry you, baby.”
He also allegedly told a court official, “Thanks for keeping it on the DL.”
While the couple tied the knot in a civil ceremony on Thursday at a New York City courthouse, the model doesn’t count herself a married woman — yet.
“I understand where the speculation is coming from, but I’m not married yet!” the 21-year-old model wrote in a since-deleted tweet.
A friend of the couple tells PEOPLE that Baldwin “feels a civil ceremony and their ‘real’ wedding are two separate things.”
Bieber proposed to Baldwin on July 7 while in the Bahamas during dinner at a local restaurant, according to TMZ, who was first to report the news.
Bieber later confirmed the news on Instagram writing, “So committed to spending my life getting to know every single part of you loving you patiently and kindly.”
“I promise to lead our family with honor and integrity letting Jesus through his Holy Spirit guide us in everything we do and every decision we make.”
Moments later Baldwin tweeted, “Not sure what I did in life to deserve such happiness but I am so utterly grateful to God for giving me such an incredible person to share my life with! No words could ever express my gratitude.”