We can see in Vannes vestiges of Roman fortifications raised in the 4th century, as well as vestiges of ramparts of the 14th and 15th centuries.
Duke of Brittany Jean II rebuilt a part of the wall in the beginning of the 14th century. After 4 sieges one after the other in 1342, walls suffered! Jean IV repaired and extended it. This was at that moment he raised his Ermine castle.
In the 15th century, they only restored towers and upper parts. During wars of religion, duke of Mercoeur erected bastions. In the 18th century, they rebuilt the current Saint-Vincent's gate, in front of the harbour.
But, by the way, where did the name "Constable's tower" come from? Well: duke of Brittany, who raised his Ermine castle near the surrounding wall, welcomed constable Olivier de Clisson, in 1387. He immediately arrested him and locked him in this tower, which was named afterwards "Constable's tower"...
Clisson, first duke's comrade in arms during the Breton war of Succession, decided to support the enemy. He became a rival, so duke Jean IV arrested him in his Vannes castle. A legend, said some people... probably, because the tower was raised after the event...