Graziano was the son of Ida Scinto and Nicola Barbella. Barbella, nicknamed Fighting Nick Bob, was a boxer with a brief fighting record. Born in Brooklyn, Graziano later moved to an Italian enclave centered on East 10th Street, between First Avenue and Avenue A in Manhattan's East Village. He grew up as a street fighter and learned to look after himself before he could read or write. He spent years in reform school, jail, and Catholic protectories. Barbella, who got occasional work as a longshoreman, kept boxing gloves around the house and encouraged Graziano and his brothers to fight one another. When he was three years old, Barbella would make him and his brother, Joe (three years his senior), fight almost every night in boxing gloves. At age 18 he won the Metropolitan A.A.U. welterweight championship. Despite the fame and money that professional fighting seemed to offer, he didn't want to become a serious prize fighter. He didn't like the discipline of training any more than he liked the discipline of school or the Army.