Justin Bieber and Hailey Baldwin got married in a courthouse on Thursday, but their actual wedding is still to come
Credit: UMV/Star Max/GC Images

Even though Justin Bieber and Hailey Baldwin got married in a courthouse on Thursday, their actual wedding is still to come.

After the model, 21, and pop star, 24, were spotted walking hand in hand to the building where marriage licenses are issued, PEOPLE confirmed the couple tied the knot, but this is only the beginning of their celebration, a source says.

“They didn’t want anyone but their families to know they got married. This wasn’t a wedding,” the source reveals, adding that the actual party with family and friends seems more important to the newlyweds than the legal act.

Justin Bieber and Hailey Baldwin
| Credit: Justin Bieber/Instagram

“They can’t wait to have a wedding,” the insider says. “But for other reasons, it was easier for them to get the legal stuff out of the way now and to get married. It was more of a necessary ceremony to make it legal. It doesn’t seem like it was a big deal to them.”

As for wedding rings to make it official? Not yet, the source tells PEOPLE. “They are not gonna wear wedding rings until after their wedding. It will be in the next few months.”

Baldwin and Bieber may have been quick about the decision, but the planning for the actual party is still ongoing. “They decided last week to get married,” the source shares. “They made an appointment [at the court] and seem happy about it. Some family members knew, but they didn’t tell friends.”

Justin Bieber and Hailey Baldwin
| Credit: Justin Bieber/Instagram

The insider adds, “They are still planning the wedding and are excited to have a celebration. They are still deciding on a wedding location.”

On Friday, a day after the legal wedding, a source told PEOPLE of the ceremony: “They went ahead and did it without listening to anyone.”

RELATED VIDEO: Hailey Baldwin Hours Before She Married Justin Bieber

The source also confirmed the couple was still planning on a bigger celebration later on, explaining, “They’re going to have a big blowout, in front of God and everyone they love.”

John Elliott - Front Row - September 2018 - New York Fashion Week: The Shows
Justin Bieber and Hailey Baldwin
| Credit: Nicholas Hunt/Getty

The couple was photographed outside the courthouse on the big day by TMZ. The outlet reported the couple appeared to be extremely emotional with Bieber telling Baldwin, “I can’t wait to marry you, baby.”

According to the TMZ, Bieber also told a court official, “Thanks for keeping it on the DL.”

The whirlwind wedding comes as no surprise, as a source close to Bieber previously told PEOPLE the pop star and his fiancée preferred quick and quiet nuptials.

Justin Bieber And Haley Baldwin Go To Dinner After Church In Los Angeles
Justin Bieber and Hailey Baldwin
| Credit: SplashNews.com

“They don’t want a long engagement and are already planning their wedding,” the source said.

“As of now, they want a small ceremony with their families. They are not planning a huge, celebrity wedding. They are getting married for love and don’t want a flashy wedding.”

The “What Do You Mean” singer popped the question to Baldwin in July while the two were on vacation in the Bahamas at a local restaurant. They had been dating for two months.

Bieber later confirmed the news on Instagram writing, “So committed to spending my life getting to know every single part of you loving you patiently and kindLY.”

“I promise to lead our family with honor and integrity letting Jesus through his Holy Spirit guide us in everything we do and every decision we make.”