Hunter Biden began dating Hallie Biden in 2017, two years after Beau Biden died of brain cancer

Hunter Biden is getting candid about his romance with his former sister-in-law, Hallie, the widow of his late older brother, Beau.

In a new interview with CBS Sunday Morning's Tracy Smith to promote his new memoir Beautiful Things, the 51-year-old son of President Joe Biden opened up about the public reaction to him dating Hallie, 46, in 2017, two years after Beau tragically died from brain cancer at age 46.

For more on Hunter Biden's book Beautiful Things, listen below to the episode of PEOPLE Every Day.

"I think people were confused by it," said Hunter. "And I understand that. I mean, I really do. To me, it's not something that is difficult to explain. Because it came out of a real overwhelming grief that we both shared. And we were together, and trying to do the right thing. And that grief turned into a hope for a love that maybe could replace what we lost."

Hunter, who split from his wife Kathleen in October 2015, five months after Beau's death, noted to Smith that his romance with Hallie "didn't work." (The relationship ended in 2019).

Hallie Biden, Hunter Biden
| Credit: Ron Sachs/REX/Shutterstock; Paul Morigi/Getty

Hunter's relationship with Hallie had mixed reactions back in 2017. While the president, 78, and First Lady Jill Biden reportedly supported the romance, many of Hunter's clients cut ties with him because of it. He also had to step down from the board of directors of World Food Program US.

"Well, I made a lot of decisions that I probably shouldn't have made," Hunter explained to Smith. "There was a lot more compassion and understanding from the people that knew me. But it was a horrible time, too."

In May 2019, Hunter wed South African filmmaker Melissa Cohen and the couple welcomed their first child, a son named Beau, in March 2020.

Hunter Biden
Hunter Biden
| Credit: Teresa Kroeger/Getty Images

Hunter's book, Beautiful Things, covers much of the journey he has taken — through drug and alcohol addiction, his tumultuous personal life, as well as his relationship with his family, including his father.

"We talk at least every night," he said of his relationship with his father. "Not only does he talk to me every night, he calls every one of my daughters and he talks to each of them every day. And he talks to me and I know that he talks to my sister."

"But, by the way," Hunter added, "he's always done that. But I'll tell you why. Because he, like me, knows what it's like not to be able to pick up the phone and talk to your son."

Beautiful Things is out Tuesday.