In 1982, Schwartz married Heather Reisman, Chief Executive of Indigo Books and Music. Schwartz has two children from his first marriage and two stepchildren from his marriage to Heather Reisman. The couple are members of the Reform synagogue, Holy Blossom Temple in Toronto. Schwartz and Reisman own the most expensive house in Toronto, valued at $28 million.