A source tells PEOPLE the relationship is "fairly recent"
Credit: Mark Davis/Getty

Hilary Duff has a new man in her life.

The Younger actress was spotted on a Tuesday night dinner date with her trainer Jason Walsh at Petit Trois in Los Angeles.

Duff – dressed in black leather pants and stilettos – and Walsh arrived at the hotspot around 8:30 p.m. on a motorcycle and stayed for nearly two hours.

“They seemed very happy and were enjoying themselves,” an onlooker tells PEOPLE.

“It’s fairly recent,” says another source about the budding relationship. “They’re casually seeing each other and this was their first real date.”

Another source says not to read too much into the dinner date.

“He is her trainer. They are friends and they go out to dinner occasionally after workouts. It’s not really a big thing,” says the source, who adds they’ve been “spending time together.”

Romance rumors first started buzzing around Duff, 28, and Walsh last summer when they partied together after the grand opening event for his Rise Nation Fitness Studio in West Hollywood.

“I’ve known [Jason] for a while, and all the trainers there,” said Duff, who has a 4-year-old son, Luca, with her ex Mike Comrie. “It’s awesome to watch him start this. It’s really changed my body.”