Overlooking the River Rhône, the place was highly strategic, since it was located between the royal estate and the Comtat Venaissin: here's Saint-André, the "New city" (Villeneuve) as they called it at that time, which is in front of Avignon. Yes, a strategical place meant wars and invasions. It had to be fortify.
So, the stronghold was raised in 1292 by king Philip the Fair, in order to protect the city and the Benedictine abbey founded in the 10th century and belonging to counts of Toulouse. And the place was occupied since a long time, with the cult of saint Casarie, a hermit who lived here, on this hill called "mount Andaon"...
In short, a garrison came soon: it will stay here for 5 centuries! Meanwhile... They raised a square tower at the end of the bridge Saint-Bénezet in Avignon, called Philip the Fair's tower. Then when popes arrived in Avignon in 1309, the two cities extended very quickly.
Popes moved in the other bank of the river, in Saint-André. So Philip the Fair's tower became pretty important, they even raised several floors in 1360. The building site of the stronghold was completed in 1372, then king Jean II the Good added two big towers on the gate.
Centuries later, they diverted the river Rhône in 1770: so, the strategical importance of the fortress just... disappeared! During the Revolution, monks were expelled from the abbey and the stronghold was abandoned.