According to tradition, Forest Hall was built as a wedding gift for Charles Barrett and his bride by his father, a prosperous farmer in town who had invested in a glass factory, a toll road, a canal system, and, most successfully, in New Hampshire's first cotton mill. The interiors are elegantly furnished, and numerous reception rooms were designed for entertaining in a cosmopolitan manner. An elaborate allée was later added to the landscape, with a flight of stone steps flanked by maples rising up the hillside behind the house and leading to an elegant summerhouse.
However, after the railroad bypassed New Ipswich, the town entered into a decline. Charles Barrett's descendants stayed on, but today Forest Hall remains essentially a relic of the Federal era. After 1887, the family used the house only in the summer-time. It was donated to Historic New England in 1950.