Today's palace complex goes back to a Niederungsburg built by Ministeriale in the 12th century. It developed from the 16th century when the three branches of the family began to expand and convert the Ganerbeburg into a castle. This resulted in three palace wings, the "Red", the "Blue" and the "Yellow Palace", which are almost completely grouped around an almost square palace courtyard (Wiki).