Qualen was born in Vancouver, British Columbia, the son of immigrants from Norway; his father was a Lutheran minister and changed the family's original surname, "Kvalen", to "Qualen" – though some sources give Oleson, later Oleson Kvalen as Qualen's earlier Surnames. His father's ministering meant many moves and John was 20 when he graduated from Elgin High School in 1920. Though he was awarded a scholarship to Northwestern University after he won an oratory contest he never attended college. In a Milwaukee Journal interview he said he needed to start working and did so with the Chattaqua Circuit. Eventually reaching Broadway, he gained his big break as the Swedish janitor in Elmer Rice's Street Scene. His movie career began when he recreated the role in the film version. This was followed by his appearance in John Ford's Arrowsmith (1931) which began a more than thirty year membership in the director's "stock company", with important supporting roles in The Searchers (1956), Two Rode Together (1961), The Man Who Shot Liberty Valance (1962) and Cheyenne Autumn (1964).