The castle is located on a strategic site. In the 12th and 13th centuries, it was the property of the des Barres family, who probably built the castle. In 1409, a charter mentioned Apremont's name for the first time: at that time, the castle belonged to Guyot de Révillon.
Then, Philibert de Boutillat, Nevers' bailiff and king of France Louis XI's paymaster bought Apremont at the end of the 15th century. The lord entirely rebuilt it, because the castle went to ruin. It was demolished during the war between king of France Louis XI and duke of Burgundy Charles the Bold.
Boutillat raised a huge quadrilateral flanked by towers with covered ways and strong curtains. Really, a nice medieval fortress! But when François de Roffignac bought the castle one century later (1602), he put up a new main building which overlooked river Allier and also added huge windows on the castle fronts.
Other alterations would come in the beginning of the 19th century: construction of stables, modification of the main building, demolition of one of the tower... Works would go on until World War II: restoration of towers, modification of the building fronts in the 15th century style...
At the end of the 20th century, new owners laid out a beautiful floral park around the castle.