Morgan was born in Wondai, Queensland, the eldest of 14 children to Dave and Ivy Morgan. From an early age he was raised by his grandparents, Bill and Eva Hopkins. After his grandfather died in 1945, he and his grandmother moved back to Scrubby Creek to live with his parents and siblings. Morgan left school at age 12 and found work cutting timber.