Jaffe was born to Heida (Ada) and Barnett Jaffe, a Russian Jewish family in New York City, New York. His mother, Ada Jaffe, was a Yiddish Actress in Odessa, Ukraine, prior to moving to the United States; his father was a jeweler. He was the youngest of four children; his siblings were Abraham, Sophie, and Annie. As a child, he appeared in Yiddish theater productions with his mother, who after moving to the United States became a prominent Actress and vaudeville star. He graduated from Townsend Harris High School and studied engineering at City College of New York, graduating in 1912. He later attended Columbia University for graduate studies. He also worked for several years as a Teacher, and then dean, of mathematics at the Bronx Cultural Institute, a college preparatory school, before returning to acting in 1915.