The Castle of Carovigno rises in an eccentric position, on the highest point of the town and, for this reason, strategically more suitable for defense. It has a triangular plan, with a square tower on the eastern vertex, a circular one on the western vertex and a "almond-shaped" one on the northern vertex.
From the top of the towers it dominates the entire coastal strip that extends from Torre Canne to Brindisi.
The first nucleus of the Castle, probably of Norman origin, is to be identified in the square tower with a strong counter-scarp adjacent to Porta Ostuni. On the south side, close to this tower, the "palatium" described in the Inventory of Maria d'Enghien of 1440 must have developed. The buildings after this date insist on the north area, incorporating both the square Norman tower both the circular one, perhaps Aragonese.
The almond-shaped tower on the north-east corner was built between the fifteenth and sixteenth centuries by the Loffreda, feuders of Carovigno in this period, as evidenced by the presence of the marriage weapon of Pirro Loffreda, walled up in the same tower. It seems that the builders of the almond-shaped tower of Carovigno were influenced by the Sienese architect Francesco di Giorgio Martini, whose presence in Puglia is attested in 1492 to supervise the construction of the strongholds of Taranto, Otranto, Gallipoli and Brindisi.