The area where the castle stands has been inhabited since at least Roman times, due to the presence of salting tanks nearby. In the 15th century, a watchtower was built.
Almádena fort is a ruin dating from the 16th century. It was built to protect the fishermen from the pirates that plagued the area. Today, the fort is open to the public and offers a lovely view of the coast.
The Governors' Castle (Castelo dos Governadores) dates from the 17th century and houses the last collection of the Seven Oratories of the Holy Place (Sete Passos da Vila Sacra) that were scattered throughout the city in the 18th century.