Olivia America Davidson was born on June 11, 1854 in Mercer County, Virginia, now Mercer County, West Virginia, to a free woman of color and a man who had been freed from slavery. Her father was Elias Davidson, who worked in agriculture. She had at least two sisters and a brother. When Olivia was still a child, her family left western Virginia because of the discriminatory treatment of free blacks and moved to Ironton, Ohio, in a free state. Some time after her father Elias died, the family moved to Albany, Ohio. Davidson continued to study at Common schools. By 1870, she was living with her sister Mary and brother-in-law Noah Elliot in Gallipolis. Her sister, a dressmaker and milliner, relocated several times before settling in Columbus, Ohio.