October 01, 2015 12:05 PM

Alec Baldwin grew up as one of six children, including actor brothers Stephen, William and Daniel Baldwin.

And earlier this week, the Concussion star, who welcomed son Rafael Thomas in June, told PEOPLE he’s "dying to have another child" with wife Hilaria. But when asked if coming from big family has made him want a large brood of his own, Baldwin said his plans are “independent of my own family.”

“Coming from my big family doesn’t necessarily mean I want to have a big family,” the actor told PEOPLE at the American Museum of Natural History in New York City, where he hosted Wednesday night’s Clio Awards.

“They were a little maddening at times, my brothers,” added Baldwin, 57.

Still, he said he’s “happier” than ever with Rafael, 2-year-old daughter Carmen and 19-year-old daughter Ireland.

Although open to the idea of an even larger clan, he said he and HiIaria, 31, aren’t rushing to meet a baby “headcount.”

“When a baby shows up that person was meant to be here. And if there is one more meant to be here, or two more meant to be here, so be it,” he said.

“And hopefully I can book a lot of commercials while I’m here today to pay for those children,” Baldwin joked about his gig hosting the advertising, design and communication awards.

You May Like