A castle in Arbon was first mentioned in 720 in the history of the monastery or princely abbey of St. Gall. It stood on or near the site of the Roman fortress Arbor Felix from 250 AD (see also Arbon Castle). After Roman rule south of the Alps ended around 400, the old fortress was abandoned. Somewhat later, a Frankish castle was built in Arbon, probably for the Frankish royal family. Around 700, Arbon, and presumably the castle as well, were owned by the Bishopric of Constance, and an ecclesiastical overseer or bailiff ruled over Arbon.
The oldest part of the present-day castle is an originally free-standing residential tower from the 13th century (1262). The tower may have been built on the foundations of an earlier building. The lower walls are up to 3.2 meters thick. Inside the castle are two Romanesque-style chimneys. Between 1515 and 1520, Bishop Hugo von Hohenlandenberg rebuilt the castle into its present appearance. The old tower was rebuilt into a U-shape. The top floor, the gables, and the hipped roof also date from this reconstruction. The castle's outer wall was shifted so that it stands directly over the northwest corner of the Roman fortress.