Joe Jonas Says of Drug Rumors: 'It's Ridiculous'

He was also upset that girlfriend Blanda Eggenschwiler was dragged into the speculation

Photo: Dan Hallman/Invision/AP

After the Jonas Brothers abruptly canceled their tour three weeks ago – and went silent as they contemplated the band’s future – reports surfaced suggesting that Joe Jonas had a drug problem.

Now the singer, 24, wishes to set the record straight, telling PEOPLE: “It’s ridiculous!”

“We’re not saying anything for two or three weeks and I’m a drug addict!”

What upset him even more is that his girlfriend, Blanda Eggenschwiler, was also dragged into the rumors.

“It’s obviously not true. I’ve never touched heroin in my life. Neither has my girlfriend,” he said. “It’s hilarious to me, but also frustrating as well. It pissed me off when it’s about my girl and even about myself.”

At one point, he said, people even approached him personally about it.

“I was walking down the street and people are asking me if they can give me a hug and how rehab is,” says Joe. “I’m like, ‘You tell me because I’ve never been.’ ”

So why didn’t he speak out right away to correct the impression?

After Nick Jonas, 21, first confessed to older brothers Joe and Kevin that he had concerns about the band – they then unanimously decided to call it quits – Joe says: “We all had to deal with it in our own way. I needed time to clear my head.”

PHOTOS: Ouch! When Celebrity Siblings Face Off

Already, that is a thing of the past. Looking ahead, Joe says, “I’m genuinely excited.”

For more on the Jonas Brothers’ decision to call it quits, pick up this week’s issue of PEOPLE, on newsstands Friday.