He made his film debut in the comedy Ants in His Pants in 1938, as an extra, produced by Ken G. Hall. At that time, he was managing a wine cellar in Bond Street, Sydney. Rafferty caught the acting bug and got another unbilled role, as one of several inept firemen in Hall's Dad Rudd, M.P. (1940).