One of the first lord of Pouzauges went with William the Conqueror in 1066, when he conquered England. When Alix de Mauléon married viscount of Thouars, Guy I Le Brun, the castle fell to this family of Thouars.
Catherine de Thouars, married the famous Gilles de Rais and lived in the castle. Her husband preferred his fief in Tiffauges.
Between 1525 and 1570, Pouzauges belonged to Claude Gouffier, royal equerry, then to the Grignon family in the beginning of the 17th century.
Then, the land fell to Charles Mesnard, marquis de Toucheprès.
Pouzauges was demolished in 1563, during wars of Religion: Protestants, led by lord de Sainte-Hermine, besieged the fortress and plundered it. During these wars, gentlemen of the area gathered here in order to plot against the Catholics.
After the signature of the Edict of Nantes, they were forced to cut themselves off; those who stayed keep gathering in some isolated places where they can pray: those places were called "Desert meetings".