The museum is housed in the "Hotel de Clerjotte", built in 1470-1480 by Louis Clergeat, taxman for Charles de France, Duke of Guyenne, and master of Île de Ré. Clergeat seems to have given his name to the building (the "House of Clergeatte") to become "Hotel de Clerjotte" today. The building was expanded in the 16th and 17th centuries, with two lateral constructions equipped with hallways and arcades. In 1684, Jean Gabaret, Lieutenant General of the Royal Navy and a Protestant, became the owner of the house. With the rise of nearby Rochefort as a key shipbuilding harbour for the Royal French Navy, Saint Martin de Ré was fortified following the designs of Vauban, and the Hotel de Clerjotte was acquired by Intendant Begon to become an arsenal. The building remained an arsenal until the 20th century. In 1929, it was designated as a historical monument, and, after some service for the Navy, was given to the city, which transformed it into a Museum.
The Hotel de Clerjotte, currently under renewal, houses the Ernest Cognac Museum, named after Ernest Cognacq, founder of the La Samaritaine Department store and benefactor of the museum. A new contemporary wings houses various displays related to the history of the island.
The garden behind the museum holds a monument with a statue of George Washington, first President of the United States. The base of the monument features a medallion representing Nicolas Martiau, a direct ancestor of George Washington. The monument was inaugurated on October 11, 2007, by the ambassador of the United States to France. The filiation between the two men is described on the monument:
Nicolas Martiau (1591 Île de Ré - 1657 Yorktown) - Jane Berkely
Elizabeth Martiau – Colonel George Reade (1608-1674)