Waldo was born in Yakima, Washington in 1919. Her mother, Jane Althea Blodgett, was a singer trained at the Boston Conservatory of Music, and her father, Benjamin Franklin Waldo, was, according to Waldo family lore, a distant cousin of Ralph Waldo Emerson. Her elder sister, Elisabeth Waldo, is an authority on pre-Columbian music and an award-winning composer-violinist who appeared in the film Song of Mexico (1945).