Brown was born and raised on a farm without running water or electricity in Bridgeton, New Jersey near the Delaware River. He learned to read early and was a voracious reader. He was fascinated by World War II and would borrow day-old papers from the next door farm to follow it. He especially enjoyed reading biographies including those of America’s founding fathers and others such as Abraham Lincoln, George Washington Carver, and Marie Curie. From his earliest years, he worked on the farm, milking cows, pulling weeds, and cleaning the stable. An enterprising youth, he involved his younger brother, Carl, in various businesses, such as growing pheasants and chickens for sale. In 1951, they started a tomato growing Business, which eventually grew to become one of New Jersey's largest, with sales of over 1,520,000 pounds (690,000 kg) a year. He later said, "farming is all I ever wanted to do with all my life. You have to know soils, weather, plant pathology, entomology, management, even politics. It's the ideal interdisciplinary profession."