The walls of Vinhais date back to the time of the Roman invasion. Later, the Suebi or Visigoths reinforced and broadened the defensive line in the context of the Muslim invasion of the Iberian Peninsula. During the reign of King Denis of Portugal, the oval fence of the village was finished, with six towers, two of them flanking the main gate and one other, also integrated in the fence, functioning as a keep and an alcove.
The castle's configuration, currently very dilapidated, would have a medieval image, in a late Gothic style. Already during the reign of King Manuel I of Portugal the degradation of the walls and of some towers was remarkable. Some bases of the old towers, a portal with a Gothic arch and a section of the barbican still remain.