Kaltenstein Castle was built in its original form by the Counts of Urach-Vaihingen and was first mentioned in a document in 1113 at the latest. After several modifications, it fell to the House of Württemberg in the 14th century and served as the seat of the princely bailiffs until the beginning of the 18th century. After being used as a barracks and military hospital, among other things, it was used as accommodation for prisoners of war and opponents of the regime in both World Wars. A memorial for the victims of National Socialism on the neighboring property testifies to this today. In 1949 the Christian Jugenddorfwerk took over the castle and managed it until 2014. According to the Monument Protection Act, Kaltenstein Castle is a cultural monument of particular importance.