When Rotrou family, future counts of Perche, became lords of that place founded in 955, they rebuilt the stony keep in the 11th century. Then Rotrou III the Great and Rotrou IV added a surrounding wall flanked by 4 metres thick towers.
They opened windows in the 30 metres high keep in the 13th century. In the 13th century, Rotrou lords died out. The castle fell to king Louis IX in 1226, who gave it to Jacques de Château-Gontier. In 1270, it fell to dukes of Brittany.
The fortress was damaged during the Hundred Years War; at that time Luxembourg family owned it and Isabel of Luxembourg (Jacques d'Armagnac duke of Nemours' widow), the new owner, restored it in 1472: it was almost ruined! Because English count of Salisbury burnt the keep in 1427, after he stepped into the castle by a huge breach (nickname the "English breach"!). The English kept Nogent until the famous Dunois (Joan of Arc's comrade in arms) expelled them in 1429...
The small castle flanked by 2 towers was rebuilt on old foundations, between 1487 and 1504. Then Nogent fell to Louis I of Bourbon in 1561, one of the Protestant leader during the war of Religion. In 1624, Condé family sold the castle to the duke of Sully, king Henri IV's minister... but he often lived here. Nogent was seized during the Revolution, and a German bombing damaged it in 1944, until city of Nogent owned it in 1948.