The castle was mentioned for the first time in 1233, in a treaty signed between local lords, as for instance a knight called Gaillard, Hugues de Larroque... and Pierre Duèze. A castle was raised, flanked by a tower where Duèze lived, while Larroque lords lived in a tower on the opposite side. All those buildings were surrounded by a big wall. Well, everyone had his own place!
In the 15th century, Larroque family raised a high tower and a main building, but... the castle fell to the Ujols, in 1546: they added wide windows in the medieval wall... Then it fell to several owners, as for instance Dufau and Pontanier du Saulon. French Revolution came and damaged Larroque castle, which was slowly abandoned. The new owners restored it and opened it to the public.