Retirement Hall was the mansion of Captain Thomas Brown, an eighteenth-century West Indian tradesman, ship owner, and slave dealer. Built in the 1760s at Pamrapaugh or Pamrapo (later Bergen Neck) in the present Greenville section, the basement of the house also served as storage for the slaves he traded.
Captain Brown was the son of English parents who had settled in Bergen County during Dutch rule. He married Anna Van Buskirk, daughter of Lawrence and Fevtie (Fytje) Vreeland Van Buskirk. She inherited an estate of slaves and property in the Pamrapo. Years after her death in 1756, Brown built Retirement Hall on her property. There he entertained guests from New York and Philadelphia.
Retirement Hall was noted for its substantial accommodations and imported English and French furnishings. It was also representative of the wealth derived from the slave trade of the time. From its location on the coast of New York Bay, Brown used his own home to store the human cargo of his maritime trade in the underground basement. When the house was razed in 1909, manacles and chains imbedded in the walls, used for securing the slaves, were revealed.
Captain Brown's only daughter (name not known) married Andrew Gautier of New York, who attended King's (now Columbia) College. They were teenagers at the time of their marriage. After their wedding at St. Paul's Church in New York City, the guests were transported to a wedding party at Retirement Hall.
It is also claimed that Prince William Henry of England, the future King William IV (r.1830-1837), was an uninvited guest at Retirement Hall. He was the third son of George III and a midshipman during the Revolutionary War. He and his companions were caught in a squall while leaving Staten Island, a Loyalist stronghold. They sought safety off the Communipaw shore and demanded food at Retirement Hall. There is no accounting of Brown at the time of the event, but it is known that he was a patriot and gave refuge to fugitive patriots in his basement. The war with England had cut into his West Indian trade.
Brown died in 1782 and his daughter died five years later. The Gautier family maintained Retirement Hall until 1829. It eventually became the Greenville Yacht Club, owned by the Lehigh Valley Railroad Company.