The Hautefage tower was built by two bishops of Agen from the della Rovere family: Leonardo Grosso della Rovere, cardinal, bishop of Agen from 1487 to 1519 and his successor, Antonio della Rovere, bishop of Agen from 1519 to 1538.
The tower was designed to serve as a residence for the bishops of Agen. The construction was probably not finished.
From the seventeenth century, the tower served as a steeple for the Notre-Dame church located below.
In the second half of the 19th century, it is considered semi-ruined. This is the state described by Georges Tholin in his article on the tower in the Agenais review of 1889.
Restoration work began in 1888, giving it the current form with a balustrade, the corner buttresses surmounted by a pinnacle and the conical slate roof added in the 1910s.
The tower was transformed into a water tower in 1957 by placing a reinforced concrete tank on the third floor. The reservoir was abandoned in 1974.
Rockfalls in 2007 lead to restoration work. The cover has been redone. The water tank was broken in 2013 and a floor was restored. The bells have been replaced on the third floor. The floors must be redeveloped at the same time as the different levels of the tower
The building is classified as historical monuments in 1883