Azay's first lord was Hugues Ridel; the lord owned a little castle here, demolished during the Hundred Years War. In the beginning of the 16th century, the estate was bought by Antoine Lesbahy.
His daughter married Gilles Berthelot in 1518, a rich and mighty Treasurer-General of the King’s finances and mayor of Tours city. He put up on the former medieval fortress basis a new castle, described as "two huge freestone buildings with machicolations, loopholes and turrets."
The works were colossal: from June to August 1518, 120 workers worked night and day in order to dig foundations, hammer the pilings and empty the land.
But we had a problem! King of France François I was intrigued by all those gorgeous castles his treasurers raised. They needed money, for that! Lot of money... Where did they find it? The king put a committee in charge of examine treasurers accounts. In 1526, the news came through: all the treasurers were guilty of embezzlement.
They took legal action against them, and against Berthelot too! His cousin, Jacques de Beaune, Lord High Treasurer was just hung in August 1527... Azay's lord was dismissed from his duties and sentenced: he owed 54 400 livres tournois. Then, he ran away to Cambrai city, leaving his castle... uncompleted!