"People are very opinionated about things they're not really aware of," Jason Aldean tells PEOPLE

Credit: C Flanigan/WireImage

Jason Aldean says there’s a lot that people get wrong about him.

He knows the cultural elite look down on his rural Georgia roots and slam him for singing about dirt roads and beer cans in koozies. He also knows people called him a cheater when his marriage to Jessica Ussery fell apart in 2013 – and that he raised eyebrows by marrying “other woman” Brittany Kerr last year. But the fact is, he’s just not going to waste his time filling in the haters.

“People are very opinionated about things they’re not really aware of,” the singer, 39, tells PEOPLE. “But at the same time I’m not going to spend all my time defending myself. I am what I am. People that know me or know my situation, they actually know.”

What he will tell you, however, is that he’s as happy as he’s ever been. And contentment, he says, is a pretty good ingredient for making music. Case in point: his latest disc, They Don’t Know.

“On this album, instead of focusing on all that other stuff that was going on for so long, everything is good and it made a world of difference for me,” Aldean says of the project. “Anytime you’re in a place where you can relax and when you’re happy, that’s going to make everything else better.”

RELATED VIDEO: Jason Aldean Talks Festivals

Don’t miss a beat of country music news, photos and videos! Click here to get all this and more in the PEOPLE Country Newsletter.

One big factor in his happiness? His new wife, who’s been keeping him company on tour. But, he admits, the two aren’t always kindred spirits musically.

“I’ll play songs for her, but it’s hard because she’s not a fan of a lot of the stuff I’m a fan of,” he says. “She’s not big into rock music and that’s one of my biggest influences. She doesn’t get that whole side of it. She’s more hip hop, R&B, pop stuff.”

For more on Jason Aldean’s new music, pick up the latest issue of PEOPLE, on newsstands Friday.

Luckily they agree on another, more important issue: “I know she would like to have at least one kid. And I’m down for that,” he says. “I love kids and being a dad is one of the best things in the world for me. If it’s meant to happen, it will. Only time will tell.”

If he does become a dad again, the singer says he’s confident that older daughters (from his previous marriage) Keeley, 13, and Kendyl, 9, would be excited about a sibling.

“Keeley’s always been a mother hen type with babies, so we’d have a built-in babysitter!” he says. “And Kendyl has always been the little sister, so for her to be the big sister, I think she’d be pretty stoked.”