Karlstein was first mentioned in 1112 as "Chadelstain." As a fiefdom, Karlstein belonged to the respective territorial princes, who granted this rule to knightly families, although no continuous dynasty established itself here. Since the castle was never owned by a single family for an extended period, it was continually rebuilt and expanded. Practically nothing remains of the original castle.
In the following centuries, the border region was repeatedly raided by enemy armies (Ottokar, Hussites, Matthias Corvinus). After the suppression of the Peasants' Revolt in 1597, the leader, Andreas Schrembser, was presumably imprisoned in Karlstein Castle.
In 1645, the Swedes besieged Karlstein Castle in vain. In 1663, it served as a refuge during the Turkish threat. In 1914, an internment camp was established in the castle, despite it being uninhabitable. The Montenegrin General Staff was also imprisoned there. After the war, the Hungarian communist leader Béla Kun was interned here. Later, the castle was used as a youth hostel, and in the 1960s, it also served as a guesthouse.
The castle is now privately owned and no longer open to the public.