Early, we find here in Crest a Roman castellum, but the tower itself dates back to medieval times. This strategic site, which watched the road linking the Alps and the Rhone valley, was mentioned for the first time in 1120 as "castle of Crest", castrum Cristoe. In 1216, Pierre de Vaux-Cernay described it as a "very noble and strong castle, well supply with soldiers and domestics". The chanson de la Croisade albigeoise ("Crusade against Cathars' song") considered Crest as an "impregnable stronghold". A true authority!
War of religion plunged the castle in the turmoil. Catholics and Protestants tried to besiege it. In January 1633, cardinal Richelieu published an edict saying that all the French medieval fortresses had to be demolish. Crest was one of them... Turned into a jail during 18th and 19th centuries, the last prisoners came out in 1852. What to do with Crest, now? The city offered to maintain it with his own money, other people suggested a quick demolition... In June 1877, the mayor listed the edifice. The debate was over! The destruction was impossible, now.