In the 1830s, Brown entered a brothel owned by Adeline Miller, a well-known New York madam. She did not stay long, however; soon Brown was running brothels of her own on Chapel and Church Streets. One brothel was partially destroyed when the neighboring National Theater burnt down in 1841. By the next year, Brown had opened a new house on Leonard Street, stocked with furniture she had salvaged from the ruined playhouse. This quickly became the most famous brothel in New York City. Fanny White, a.k.a. Jane Augusta Blankman once worked in Brown's brothel.