Retirement Hall (ca. 1760) was the mansion of Captain Thomas Brown, an eighteenth-century West Indian tradesman, shipowner, and slave dealer, and representative of the wealth derived from the slave trade of the time.
Captain Brown was the son of English parents who had settled in Bergen during Dutch rule. He married Anna Van Buskirk, daughter of Lawrence and Fevtie (Fytje) Vreeland Van Buskirk. She inherited an estate of enslaved persons and property at Pamrapo (later Bergen Neck) on the New York Bay coast in the present Greenville section. Years after her death in 1756, Brown built Retirement Hall on the property.
Retirement Hall was noted for its substantial accommodations and imported English and French furnishings. There, Brown entertained guests from New York and Philadelphia. He also used his home to store the human cargo of his maritime trade in the underground basement. In 1909, manacles and chains used for securing the enslaved were found embedded in the walls when the house was razed.
Captain Brown's only daughter (whose name is unknown) married Andrew Gautier of New York, who attended King's (now Columbia) College. They were teenagers at the time of their marriage. After their wedding at St. Paul's Church in New York City, the guests were transported to a wedding party at Retirement Hall.
It is claimed that Prince William Henry of England, the future King William IV (r.1830-1837), was an uninvited guest at Retirement Hall. He was the third son of George III and a midshipman during the Revolutionary War. As they were leaving Staten Island, a Loyalist stronghold, Prince William Henry and his companions were caught in a sudden storm. They sought safety off the Communipaw shore and demanded food at Retirement Hall. There is no accounting for Brown at the time of the event, but it is known that he was a patriot and gave refuge to fugitive patriots in his basement. The war with England had cut into his West Indian trade.
Brown died in 1782, and his daughter died five years later. The Gautier family maintained the Retirement Hall until 1829. It eventually became the Greenville Yacht Club, owned by the Lehigh Valley Railroad Company.