Fans said goodbye to another hopeful Tuesday night as the battle for the coveted mirror-ball trophy continues

Credit: Adam Taylor/ABC/Getty

Tavis Smiley has a bone to pick with the producers of Dancing with the Stars.

The talk-show host, who on Tuesday became the second celebrity to get booted from the competition, said there was no way he could fit in rehearsal time with his latest book tour. And the producers knew it, he insisted.

“Producers shouldn’t sign up people who can’t commit the time when they are told up front that the person is going to be on a book tour,” Smiley told PEOPLE immediately after the show. “We gave them the entire book-tour schedule in advance! This is not about sour grapes. It’s about saying I think the fans deserve to have stars on this show who have the time to commit to learning the dances so [partner Sharna Burgess] and other professionals don’t waste their time trying to do something that’s just humanly impossible.”

Smiley and Burgess only had 10 hours last week to prepare for Monday’s cha-cha while Smiley promoted his latest book, Death of a King: The Real Story of Dr. Martin Luther King Jr.’s Final Year. The lack of rehearsal time showed: Though the duo didn’t earn the lowest score (that went to Michael Waltrip, who earned a 24 out of a possible 40), Smiley got the second lowest score – a 28. Judge Len Goodman told Smiley he lacked polish, while Bruno Tonioli said he needed to be lighter on his feet.

“I think part of what fans may have been saying tonight was, ‘Why keep Tavis around if he doesn’t have the time to really dedicate to this?’ ” Smiley said. “Some people thought that Michael didn’t have his best night last night. And he wasn’t the best. His scores were lower than ours, that’s the fact. But I think the fans can see that Michael is here, and he has time to actually commit to this. He’s not on a book tour.”

“I asked for the spring,” Smiley continued. “With all due respect to the producers, when they came after me, I said to them specifically I’m going to be on a book tour in September. Everybody on this show will tell you it’s harder than it looks. You have to spend time perfecting. I’ve been a talk-show host for 20-something years and I’m still getting better at it. But you can’t perfect this in eight or 10 weeks.”

Burgess admitted it got pretty stressful following Smiley around the U.S. while trying to squeeze in practice time. She visited Smiley in five cities to prepare for Monday’s performance.

“At that point, you have to steal five minutes wherever you can elevators, hallways, waiting backstage before he does his lectures,” she said. “It was a very intense process. I definitely had anxiety the whole week. But we made it work the best we can. At the same time, I’m incredibly proud of what we were able to do in 10 hours.”

So what’s next for Smiley? He has two more weeks of promoting Death of a King before preparing to return for the DWTS finale.

“I told Shar if we miraculously got through two more weeks of this, we could really drill down,” Smiley told PEOPLE. “But the clock ran out on us. It’s like a football game. Sometimes you are making a comeback, and the clock just runs out. I did the best I could with the time we had. No regrets!”

Want more stories like this?

Sign up for our newsletter and other special offers:

Thank you for signing up!