Over the centuries, the castle - initially a castle - had experienced an eventful history and many owners. This also applied to the village and its subjects, who were always dependent on the lord of the Weisendorf manor. Immediately after the Thirty Years' War, work began on rebuilding the castle. In 1689 it was given its present form with four corner towers and an inner courtyard, based entirely on the model of Seehof Castle near Bamberg. In 1784 the entire property passed to Count Rumerskirch. According to the chronicle, he had the castle moat filled with 400 loads of sand to the level it is today. This lost its character as a moated castle. In 1813 the Imperial and Royal Austrian Grand Ducal Tuscan chamberlain Franz Ludwig Freiherr von und zu Guttenberg owned the manor of Weisendorf. The castle remained in the possession of this family until after the Second World War. However, until 1945 it was only used as a summer residence.
In 1957, Baron Karl Theodor von und zu Guttenberg donated the castle and outbuildings to the French secular order "Notre Dame die Vie".