Updated September 26, 1994 12:00 PM

AS ELEGANT AND DISCREET AS ITS owner, Oliver Hoare’s Ahuan Gallery is a curiously uninviting place these days. On posh Eccleston Street in London’s Belgravia, the gallery where Hoare, 49, sold Islamic art to clients including Queen Noor of Jordan and the Sultan of Brunei looks deserted: No one answers the door, and the reception desk is unattended. A month after London papers broke the story that the dashing Hoare had received over 300 anonymous phone calls believed to be from the Princess of Wales, the only artifact in the display window is, symbolically, a suit of armor.

Since being pegged as the man with whom the princess is obsessed, Hoare has found himself under attack from all sides. Fleet Street’s finest have been camping out at the $3.5 million house on Chelsea’s fashionable Tregunter Road where he lives with wife Diane, 46, and their children Tristan, 17, Damien, 15, and Olivia, 12. Tabloids are gleefully reporting that, instead of being a family man who merely offered a sympathetic shoulder to Diana when her marriage was foundering, Hoare is a compulsive charmer who had a fiery liaison with a Turkish-born socialite. And behind the scenes, insiders are hinting that his relationship with Di, 33, may not have been platonic. “The consensus is that they definitely had an affair,” says one seasoned Di-watcher. “No question about it. Diana became completely besotted with him.”

Palace-watchers also claim that Prince Charles and longtime lover Camilla Parker Bowles (also a close friend of Hoare’s) knew of the alleged affair and Di’s phone calls six months before the story emerged in the press. In the words of Daily Mail columnist Nigel Dempster, Charles’s knowledge of the “burgeoning friendship” between Di and Hoare made Charles feel “less constrained in making his admission… that he had broken his marriage vows”—a confession offered in an interview with freelance broadcaster Jonathan Dimbleby on ITV last June.

Hoare’s professional life has come under scrutiny as well. In recent weeks, London has learned that his gallery has been losing money heavily—as much as $700,000 last year—and that he once sold a valuable page from an early Koran that proved to have been stolen. Though colleagues who will speak on the record characterize his business dealings as “utterly beyond reproach,” it came to light following a 1989 burglary at his gallery that Hoare had overinsured some antiques for more than 10 times their real value.

On Sept. 7 the tale became still more tantalizing when nude photos of Charles, who is close to both Hoare and his wife, were published by the German paper Bild Zeitung. As it happened, Charles was on a solitary holiday at a chateau near Avignon, France, owned by Diane Hoare’s mother, Baroness Louise de Waldner, when paparazzi caught him toweling off in his bedroom. Besides creating an uproar over the intrusion, the incident demonstrated just how close the two families are—and how little Charles seemed to mind that his wife has been under Hoare’s spell. Says one royal-watcher: “The fact that he was prepared to stay at the villa while the story was unfolding is evidence that he’s still friendly with Oliver. I don’t think Charles really cares if [Hoare] was having an affair with Diana or not.” Adds Lady Colin Campbell, author of Diana, the Princess Nobody Knows: “Why would Charles mind? He has nothing to do with the woman. The fact that he knew of the affair is a sign of his complete indifference.”

In the wake of these embarrassments, Hoare retreated to the safety of his antique-filled home, where he vents his frustrations in what he calls his screaming room—a soundproof haven where he copes with stress by yelling at the top of his lungs. He emerged from his house briefly on Aug. 28 for a stroll with son Damien, but politely declined to answer questions from the press.

As friends tell it, the man whom Diana chose as a confidant has always been known for his discretion. “He’s an old-fashioned gentleman,” says Dale Egee, proprietor of a London gallery called Egee Art Consultancy. “He has a very scholarly background and a retiring, even vulnerable manner that is very appealing to women.”

The son of Reginald Hoare, a civil servant, and his wife, Irina, Hoare studied at the Sorbonne after graduating from Eton in 1963. Fascinated with Iran, he found a mentor in Princess Hamoush Azodi-Bowler, who invited him to stay at the Tehran mansion where she played host to a circle of artistic young aristocrats. (It was there that Hoare met business partner David Sulzberger, whose family owns The New York Times.) “Everybody loved him,” Azodi-Bowler told the Daily Mail. “He played the guitar and sang beautifully…[at parties] he would dedicate different songs to different women.”

Described by a friend who spoke to the Daily Mail as an “extravagantly pretty” young man “who would turn the heads of both men and women,” Hoare began to move in ever more exalted circles. After returning to London he set up a gallery that attracted customers including Rudolf Nureyev and in 1974 began wooing Diane de Waldner, a French oil heiress whose mother is close to the Queen Mum.

It was eight years after the Hoares’ 1976 wedding that Oliver met Charles at a Windsor Castle house party. The prince struck up a friendship with the worldly Hoare, who became a frequent visitor at Highgrove, Charles’s home in Gloucestershire. By the late 1980s, Hoare had also befriended Diana, who shared his love of ballet.

That relationship seems not to have become close, however, until 1992, when Hoare helped to cushion the breakup for both Charles and Di. By that time, his own marriage was still recovering from his six-month fling with ex-beauty queen Ayesha Nadir, whose estranged husband, clothing tycoon Asil Nadir, had been a client. On Aug. 28, Ayesha, now divorced, told the News of the World that the affair began in 1986, when Hoare offered her solace after she learned that Nadir’s mistress had borne him two children. Traced to her villa on the Bosphorus, Hoare’s ex-lover (who reportedly left him after concluding that he would never divorce Diane) told The Evening Standard, “Oliver is such a refined man. It is hard to resist his charms.”

Di, too, apparently found Hoare difficult to resist. “I think she fell in love with Oliver, maybe because she’s never been loved properly,” Ayesha told the News of the World. Though it is unclear just when—or whether—the two had a physical relationship, Diana seems to have been infatuated with Hoare by September 1992, three months before the Waleses separated and the anonymous calls began. Although Diana reportedly was silent whenever Hoare answered, she is said to have shouted “a stream of abuse” when Diane picked up the phone.

Although most Di-watchers feel that she made the calls simply to hear Hoare’s voice, Campbell believes that Diane may have been the target. “[Diana] made those calls to taunt her because she’s jealous,” says Campbell. (That notion was supported by recent reports that, for years, Parker Bowles received silent calls that she claims were from her royal rival.) For her part, Diane reportedly believed that the relationship between Di and her husband was getting out of hand; Dempster said recently that she told Charles that Hoare had “had an affair” with Diana.

In any case, it was Diane (whose money reportedly is helping to offset her husband’s business difficulties) who demanded that Hoare ask the police to investigate. Still, he declined to press charges after he discovered last January who his caller was, and he apparently did not abandon the princess. According to The Sun, Hoare (who was still serving as a liaison between the Waleses) called Diana “at least 15 times” in the days following a March 14 dinner with friends at a Chinese restaurant, after which he returned with her to the palace. “A royal source” told the paper that one call last April “started at around 8:30 at night and went on for close to [90 minutes]. Who knows what was being talked about, but they obviously had plenty on their minds.”

At the moment, the charismatic Hoare seems to be hoping that lying low will solve at least one of his problems—that of being stalked by a rowdy crowd of newsmen. Whether it can quell Diana’s obsession remains to be seen. Last month, the. News of the World reported that the distressed princess rang Hoare’s home seven times during a four-day period in late August—after the tabloids had reported that she was hounding him. Telling the housekeeper that she had to “speak to [Oliver] urgently,” the princess reportedly hung up when she was told it was impossible. In the weeks to come, it seems, that “screaming room” may come in handy.