"It's kind of hard to tell what people think about you," the Today co-host said on Watch What Happens Live

Here are a few things people get wrong when they ask Jenna Bush Hager about her life when her parents were the president and first lady: No, she didn’t live in the White House (she was already in college) and, no, that spotlight of international fame didn’t worsen her work ethic.

The 38-year-old Today show co-host was appearing on Watch What Happens Live with Andy Cohen on Wednesday night with Hoda Kotb and was asked by a viewer, “What do you think is the biggest misconception of being the child of a U.S. president?”

“Well people ask that question some,” Hager said. “It’s kind of hard to tell what people think about you … but I do think it’s hard: People don’t really know my parents, know who they are as my parents.”

Kotb, 55, jumped in to offer her take.

“I think Jenna, by the way, I think you’re one of the most down-to-Earth people,” she said. “And we work with her every day and you never get the feeling that you’ve lived any other life than the kinds of lives we’ve lived.”

“Well we didn’t live in the White House. So people will always be like — ‘What was it like there?’ ” Hager said. “[But] I was in college, so I lived in Austin.”

Jenna Bush
Jenna Bush Hager
| Credit: Jeremy Smith/imageSPACE/Shutterstock

RELATED VIDEO: Hoda Kotb and Jenna Bush Hager Start Intermittent Fasting

Hoda Kotb and Jenna Bush Hager
Hoda Kotb (left) and Jenna Bush Hager on Today
| Credit: Nathan Congleton/NBCU Photo Bank/NBCUniversal via Getty

“It’s hard to tell what people think about me, but I think probably, maybe, that I’m not a hard worker,” Hager said. “I try to fight against that a little.”

And how: Last fall, PEOPLE followed along for one bustling afternoon that saw Hager shuttling from the Today set back home to her three children, baby son Hal and two young daughters, while juggling obligations for her kids’ and her then-ongoing book tour with twin sister Barbara.

“Whenever anybody walks in, we go, ‘Welcome to the nut house.’ Who says that? I do,” she joked to PEOPLE then.

Elsewhere on Watch What Happens Live on Wednesday, Hager (again) addressed the controversy last year over Ellen DeGeneres being friends with her father, former President George W. Bush, and she and Kotb talked about Jessica Simpson‘s new memoir and what ex-husband Nick Lachey and his wife, Vanessa, had to say.