The area where the castle stands has been inhabited since at least Roman times, due to the presence of salting tanks nearby. In the 15th century, a watchtower was built.
In the early 17th century, under Philippine rule, the government began a programme to build several fortresses along the coast to defend it from attacks by pirates and corsairs. A fortress was planned to be built at the mouth of the Arade River, although initially no agreement was reached on which side of the mouth it should be built. The Italian military engineer Alexandre Massai argued in favour of the left bank, where Vila Nova de Portimão could be better defended, and so the fortress was built on this side.
It was only after the restoration of independence in 1640 that plans began to be made to build the castle on the right bank, as a way of protecting the coast from Spanish attacks. The works took place between 1643 and 1644, on the site of the old watchtower, known as Ferragudo Castle, which was no longer considered strategically effective. In fact, it served not only to protect but also to control the mouth of the Arade, and was a customs registration point for ships sailing along the river to the city of Silves. In 1654, the castle underwent a process of refurbishment.
The building was gradually abandoned and was already in ruins by 1669. However, the castle was still subject to extension works in the 18th century, and was only slightly damaged by the 1755 earthquake, due to the solid rock on which it was built.
In 1821, two batteries were still in operation on the highest part of the castle, both with gun ports. By 1861 it was once again in ruins, and in 1892 it was leased to the writer and diplomat Joaquim José Coelho de Carvalho, whose family had bought the building in 1896. Joaquim de Carvalho carried out renovations at the beginning of the 20th century, which gave it the appearance of a romantic castle. Some time later, it became the property of the politician Francisco Vieira Machado. The monument did not undergo any major renovations in the 20th century. The changes that were made to its interior were mainly used to transform it into a holiday home, although much of the building remained in a state of disrepair. In 1976, the Ferragudo Parish Council asked for the fort to be ceded to it, so that a cultural centre could be set up inside.