Graham was born on March 6, 1951, in the Whitesboro section of Middle Township, New Jersey, the son of Mary Jacobs Graham and Stedman Graham, Sr. He is one of six children. He stands 6 feet 6 inches tall. He received a bachelor's degree in social work from Hardin-Simmons University in 1974 and a master's degree in education from Ball State University in 1979. Graham played college basketball at Hardin-Simmons.