Taubman was born in Pontiac, Michigan, on January 31, 1924, to Jewish immigrants Fannie Ester Blustin and Philip Taubman. His parents came to the United States from Białystok, in northeastern Poland. His mother was his father's second cousin. Philip took a job with the Wilson Foundry Company in Iowa, transferred to Pontiac in 1920, became a fruit farmer, then began developing commercial real estate and custom homes.