The castle was built on top of a Roman villa which, in turn, was probably built on top of an Iberian settlement.
During the Middle Ages it was owned by different feudal lords: Simon de Geronella, Jaume de Cornellà, Bernat Pallarès (Barcelona), his son Pere Pallarès and, finally, Berenguer de Cruïlles, an important feudal lord of Empordà, who bought it in the previous owner for 20,000 Barcelona sous. In 1484, all the domains of Sant Esteve Castle and the town of Palamós passed into the hands of Galceran de Requesens, first count of Palamós.
During the War of the Remences it was demolished almost in its entirety and was abandoned until in the 16th and 17th centuries it was converted into a farmhouse. The castle becomes a strategic point of surveillance and advanced defense of Palamós and its surroundings. At the same time, it became a farmhouse. During the Modern Age, agricultural exploitation was combined with control of the coast, especially to alert the population of raids by pirates and corsairs from North Africa.
The old castle remained as a farmhouse until the middle of the 20th century, when it was definitively abandoned. From that moment, it suffered a slow but progressive degradation. Its situation, near the sea, subject to inclement weather, and also away from the most frequented centers, were determining factors for its decline.