The castle was originally built as a monastery priory of the Benedictines of Weingarten in 1654 by the Vorarlberger Michael Beer. In 1802 it was dissolved as a monastery and came in the course of secularization in 1806 in the possession of the Kingdom of Württemberg. Under King William I of Württemberg took place in the years 1823 to 1830 according to the plans of the then court architect Giovanni Salucci the conversion to the royal summer residence. The monotonous south wing of the monastery was adorned by a two-storey loggia built in the middle with a semi-storey on top, creating a castle-like appearance. In the west wing were the royal private rooms, while in the eastern wing the meeting rooms and the guest rooms were furnished.
In air raids, the castle was badly damaged in 1944 by incendiary bombs. The three wings burned down to the vaults of the ground floor and the stone outer walls, and the outbuildings were almost completely reduced to rubble. The construction of an emergency roof in the winter of 1948/49 saved the ruin from further decay. In the years 1951 to 1965 was a monumental reconstruction of the castle. From the economic section, only the south wing was restored, the other three wings were demolished in 1958.
Today, the ducal family lives in the west and south wings of the former residence; The ducal administration, the Hofkammer, is housed in the east wing. Today, the castle is the residence of Friedrich Duke of Württemberg. An inside inspection is not possible.