Karle was born in New York City, on June 18, 1918, the son of Sadie Helen (Kun) and Louis Karfunkle. He was born into a Jewish family with a strong interest in the arts. He had played piano as a youth and had participated in a number of competitions, but he was far more interested in science. He attended Abraham Lincoln High School in Brooklyn, and would later join Arthur Kornberg (awarded the Nobel in Medicine in 1959) and Paul Berg (a winner in Chemistry in 1980), as graduates of the school to win Nobel Prizes. As a youth, Karle enjoyed handball, ice skating, touch football and swimming in the nearby Atlantic Ocean.