In the 13th century there was a castle on today's grounds of the JVA Waldheim. In 1404 the castle was rebuilt into an Augustinian monastery. With the dissolution of the monasteries, Georg von Carlowitz received the property. Elector Christian I of Saxony bought the former monastery grounds from him in 1589 and converted it into a hunting lodge. August I of Saxony (August the Strong) had the castle grounds converted into a penitentiary, poorhouse and orphanage. From 1830 it became a pure prison. In 1868 the castle was demolished except for the castle church and the gatehouse. In the years that followed, several extensions and new buildings were built. The most well-known inmate was Karl May. In the JVA there is a small museum on the history of the facility. Visitors to the museum must register in advance via the internet (minimum 15 people)