Anna Nicole Smith's attorney and confidant played a controversial role in her life

Howard K. Stern was known to fans of E!’s Anna Nicole Show as Anna Nicole Smith’s lawyer, confidant and occasional sparring partner. Then in September, he told Larry King he was also her lover – and the father of her infant daughter.

But who exactly is he?

A graduate of UC-Berkeley and the UCLA School of Law, Stern, 38, cofounded a firm that handled Smith’s modeling and show business contracts in the mid-1990s.

When her husband J. Howard Marshall II died in 1995 and she entered into litigation with his family over his $1.6 billion estate, Stern helped out. “Howard assisted capably as a resource,” her lead counsel, Philip Wendel Boesch, Jr. told PEOPLE last year. “He presented Anna’s direct examination in the court trial. I did the closing argument and the cross-examination of witnesses.”

By the time Stern’s law firm dissolved about five years ago, he had become more involved in Smith’s affairs, negotiating her E! show and even appearing with her in the 2003 B-movie Wasabi Tuna.

It was clear to those who knew Smith that Stern was indispensable to her. “She relies on his opinion for everything and he takes care of every aspect of her life,” one source told PEOPLE in September. “He was always there for her, even when she was dating other guys.”

Added the source: “He fell in love with her the moment he met her. He has always been madly in love with her.”

After the death of Smith’s son Daniel in her hospital room just days after she gave birth to daughter Dannielynn, Stern himself admitted as much, telling Larry King on Sept. 26 that he was Dannielynn’s father and was hoping to marry Smith. “I’ve loved her for quite some time and hopefully the feeling’s mutual,” he said. “At some point, we will. Right now we have to somehow get through what we’re going through.”

Two days after that interview, Stern and Smith held a non-binding commitment ceremony in the Bahamas.

Some observers wondered if Stern was taking advantage of Smith during her time of grief, but in November Smith said in an interview with Entertainment Tonight and The Insider that they’d been involved for years – and she’d made the first move. “I wanted more than a lawyer relationship,” she said. “I just kind of attacked him and that was it. … He was the shy one.”

But members of Smith’s family questioned Stern’s motives and wondered if he’d had a role in Daniel’s death. Smith’s mother, Virgie Arthur, told CNN’s Nancy Grace: “There was only three people in that [hospital] room. … Now if Howard Stern marries [Anna] and she ends up dead, then who does the money go [to]?

The night of Smith’s death, her sister, Donna Hogan, was even more blunt. “Don’t even get me started on that man,” she told Larry King on Thursday. “I don’t have really good feelings about him right now. I have two family members that have died … and it’s like he just stood by and let it happen.”

Visit for continued coverage of Anna Nicole Smith as the tragic story continues to develop.