Calfa was born on December 3, 1939 in Brooklyn, New York, and was raised in Ozone Park, Queens and later West Hempstead, Long Island. According to his official biography, Calfa was originally interested in pursuing a career in the fine arts, but diverted his interest to acting after seeing Rebel Without a Cause. He promptly dropped out of high school to join a theater workshop (later finishing his degree through night school), eventually joining both the Actors' Equity Association and the Screen Actors Guild. Before transitioning to film, Calfa worked heavily in theater, appearing in many Off-Off-Broadway productions and having performed on Broadway in 1965 in Mating Dance and in 1971 in Lenny.