Fortas was born in Memphis, Tennessee, the son of Rachael/Ray (née Berson) and Wolf/William Fortas, a cabinetmaker. He was the youngest of five children. His parents were Orthodox Jews. His father was born in England, to parents from Russia, and his mother was Russian-born. Fortas acquired a lifelong love for music from his father, who encouraged his playing the violin, and was known in Memphis as "Fiddlin' Abe Fortas". He attended public schools in Memphis, graduating from South Side High School in 1926. He next attended Southwestern at Memphis, a liberal arts college now called Rhodes College, graduating in 1930.