Foulques Nerra besieged the castle (which belonged to the counts of Blois) in 1026. Anjou region was at that time an English property, since the marriage of Geoffroy Plantagenet's son, Henri II, count of Anjou. But king of France Philip Augustus get Saumur back, which became a royal property. In 1360, Louis I of Anjou embellished the façade with Gothic elements. René I the Good continued the building works and replaced the lead roof with slate.
Then Saumur fell to the Protestant Philip de Mornay: this one became governor of the city. He asked the Italian artist Bartolomeo, in 1590, to raise a kind of bastion next to the castle. He also entirely redecorated the apartments. The legend said the governor had one of the richest libraries in France, and he wanted to give it to the Protestant church of Saumur!
De Mornay left his fortress in May 1621. As soon as he left, king's guards plundered the castle and destroyed all the precious books... Then the castle was abandoned. In the 18th and 19th centuries, Saumur fortress became a jail. The city of Saumur owned it in 1906 and opened it to the public.