"We didn't really know where we wanted to go, but we knew we weren't where we wanted to be," Ami Brown tells PEOPLE
Credit: Ture Lillegraven

Bill and Ami Brown weren’t always the Alaskan wilderness experts fans have fallen in love with on Alaskan Bush People. In fact, they were raised in a far different climate.

Hailing from small towns in North Texas, Bill, 65, and Ami, 52, wed in 1979 and started a life in nearby Fort Worth.

“We settled down and tried to do everything ‘right,’ ” Bill says in the latest issue of PEOPLE.

He started a small plumbing company but says he came home one night and told Ami, “I don’t know if I can do a 9-to-5 the rest of my life.”

Luck for him, Ami felt the same way.

“We didn’t really know where we wanted to go, but we knew we weren’t where we wanted to be,” she says.

The future Discovery Channel reality stars traveled around the “lower 48” states until 1983, when they decided to sell their truck and Bill’s tools and buy tickets on a ship heading to Alaska with their first two sons, Matt, then 3, and Joshua (who goes by Bam, a shortened version of the middle name given to him in honor of his great-grandfather’s love of the Flintstones character Bamm-Bamm), then 1.

As they approached their first winter in Alaska, the Browns accepted a friend’s offer to stay in his trapper shack on an island approximately 50 miles from the city of Wrangell.

“We ended up stranded on that island for 18 months,” says Bill. “The Good Lord just made us slow down. After we realized we weren’t going to die, we fell in love with everything.”

Eventually, a skipper came upon the family and offered them a ride to the nearest town.

“[The skipper] said, ‘You can have your lives back,’ ” says Bill. “But Ami and I looked at each other, and we knew we didn’t want that.”

Instead, the couple decided to raise their growing brood – which now includes seven kids: sons Matt, 33, Bam, 31, Bear, 28, Gabe, 25, and Noah, 23, and daughters Snowbird, 21, and Rain, 12 – in the Alaskan wilderness.

A few years ago, a producer came across Bill’s self-published autobiographies and pitched what has become Alaskan Bush People, which pulls in an audience of almost 5 million people according to a rep for Discovery Channel.

“It can be rough out here at times,” Ami says of the dangerous conditions under which she has raised her family. “But we wouldn’t have it any other way.”

Season 3 of Alaskan Bush People airs Wednesdays (9 p.m. ET) on Discovery Channel.

For more on the Brown Family’s life in the wild, pick up PEOPLE’s Sexiest Man Alive 2015 issue, on newsstands Friday