Anatole Broyard was born in 1920 in New Orleans, Louisiana, into a mixed-race Louisiana Creole family, the son of Paul Anatole Broyard, a carpenter and construction worker, and his wife, Edna Miller, neither of whom had finished elementary school. Broyard was descended from ancestors who were established as free people of color before the Civil War. The first Broyard recorded in Louisiana was a French colonist in the mid-eighteenth century. Broyard was the second of three children; he and his sister Lorraine, two years older, were light-skinned with European features. Their younger sister, Shirley, who eventually married Franklin Williams, an attorney and civil rights leader, had darker skin and African features.