Schwartzman and his first wife, Judith Deborah Feldman (m. February 5, 1958 – August 11, 1980; divorced) had two children, John and Stephanie. Schwartzman and his second wife, Actress Talia Shire had two sons, Jason and Robert. Jack Schwartzman died of pancreatic cancer on June 15, 1994 in Los Angeles, California.