"Harry also vacationed with friends, so it wasn't just a romantic vacation for Harry and Kendall," a source tells PEOPLE
Credit: Jeff Kravitz/AMA2015/FilmMagic; Jason Merritt/Getty Images

Kendall Jenner and Harry Stylesspent quality time together over the New Year’s holiday weekend, but a source says speculation that they are a couple is premature.

“Harry also vacationed with friends, so it wasn’t just a romantic vacation for Harry and Kendall,” the source tells PEOPLE.

The One Direction member, 21, invited Jenner, 20, to join him on vacation, and she flew by private jet from Los Angeles. The pair, who vacationed in Anguilla in the Eastern Caribbean over the weekend, first met in L.A. about two years ago.

“They had fun then, but things never got serious,” the source says of why things didn’t become more serious back then.

The festivities over the holiday included partying on a yacht, where they were joined by Ellen DeGeneres and Portia de Rossi.

“He has thought all along that Kendall is hot and was sort of just waiting for a chance to spend more time with her again,” the source says of Styles. “New Year’s worked out perfect, and he loved it.

Jenner returned to L.A. on Saturday, but the source says it’s not goodbye for good.

“They will stay in touch and plan on meeting up soon again,” the source says.

A source previously told PEOPLE that the stars were quite flirtatious on the yacht.

“There has been a lot of flirting, touching and snuggling going on,” a source tells PEOPLE. “They have acted like a couple. They look very cozy and happy.”

The model and singer were first linked in January 2014, when they were seen going out to dinner and hitting the slopes at Mammoth Mountain in California last year, but they never confirmed the relationship.