Lee was born and raised in Rye, New York. His parents, Annette (Franks), a Teacher, and Marvin Lee, an electrical Engineer, were children of Jewish immigrants from England and Lithuania. He graduated from Harvard University in 1952 and then joined the U.S. Army for 22 months. After being discharged from the army, he obtained a master's degree from the University of Connecticut. In 1955 Lee entered the Ph.D. program at Yale University where he worked under Henry A. Fairbank in the low-temperature physics group, doing experimental research on liquid He.