The primitive castle was destroyed, sorry... Completely burnt during battle of Villersexel, which took place in the neighbouring plain: French and Prussians fought there, on January 9th and 10th 1871. The French won... it was the only French victory in this war opposing France and Prussia!
So, city of Belfort was, in Eastern France, the only place which remained French. Belfort, bravely defended by famous colonel Denfert-Rochereau... The French decided to send general Charles Bourbaki, leading an army of 12 000 soldiers. To save Belfort from enemies! But on their way, they came up against Prussians in the plain of Villersexel! The battle lasted 2 days.
Prussia lost 4000 soldiers, not to mention the prisoners. What about the castle? It fell into the hands of Prussians. The French had to turn them out! They shot at them and they finally left: but they started a fire... The current castle dates back to 1885, re-raised after the battle.