A fortified Celtic hilltop settlement can be traced back to around 550 BC on Sophienberg. In the 12th century, Counts Ulrich and Poppo von Blassenberg built Culmberg Castle. In 1494, Frederick II, Margrave of Brandenburg-Kulmbach, had a watchtower erected there. Nikol von Heerdegen built a knight's castle on the hill in 1513. In 1614, Heerdegen sold the manor to Margrave Christian's first wife, Marie of Prussia, who had the castle restored. In 1662, Margrave Christian Ernst gave the estates on the hill to his young wife, Erdmuthe Sophie. Between 1663 and 1668, she had the already dilapidated castle converted into a new palace, Sophienburg. Tradition has it that the White Lady haunts this building. It was no longer inhabited by 1687, and from 1724 onward, it was no longer administered. It was left to decay, and only remnants of the foundations remain.
The Sophienberg exerted a great attraction on Richard Wagner, who first visited it with his wife Cosima and Malwida von Meysenbug in 1872 during his time in Donndorf.