The oldest part of the castle called Ulsitz dates back to the 13th century. It was probably built by the burgraves of Dohna and belonged to the old Bohemian Bunzlau district. During the Hussite wars it was conquered and rebuilt several times.
In 1562, the imperial councilor Georg Mehl von Strehlitz (Jiří Mehl ze Střelic; † 1589) acquired the Grafenstein castle and dominion from the brothers Albrecht and Rudolf von Dohna (Donin) for 30,000 Rhenish Floren. He converted Grafenstein Castle into a Renaissance palace, while retaining the fortifications. He later sold both to his relative Ferdinand Hoffmann, Freiherr von Grünbühl (Ferdinand Hoffmann z Grünenbühlu) for the same price.
During the Thirty Years' War, the castle was conquered by the Swedes, who used it as a military hospital and supply base for other military operations in the area. Around this time, the castle and lordship passed to the Trauttmansdorff noble family, from whom the Clam-Gallas acquired it in 1704. They had extensive possessions in northern Bohemia and connected the castle and dominion of Grafenstein with their dominion of Friedland. In 1782 they rebuilt the castle and at the same time renewed the interior.
Further restorations took place in 1818 and after a fire in 1843. Before 1838, Christian Christoph Clam-Gallas acquired the altarpiece "Maria im Grünen" for the castle chapel of St. Barbara, which was probably a copy of the first work by the painter Joseph von Führich.
After World War II the castle was confiscated and nationalized by Czechoslovakia. Some buildings were initially used by the army as a defense base, barracks and kennels. In the years that followed, it was partially left to decay.
After the Velvet Revolution in 1990, extensive restoration work was carried out on the castle, the overall appearance of which has been preserved to this day.
Numerous collections, old cellar rooms with exhibitions, the war tower with a viewing platform and the castle chapel are accessible. In the deep subterranean rooms there is a dungeon that can also be visited.
In the courtyard, top-class benefit concerts with soloists from the Prague National Theater and the local soloist Luďka Vele take place every year. The money goes towards the restoration of the castle.