The castle was erected in the 14th century on the left bank of the Barycz River. In 1560, within the castle walls, a four-storey defensive residential tower was erected, built of stone, with an added turret with stairs, topped with battlements.
In 1579 and 1605, the castle was consumed by fires, causing serious damage. During the Thirty Years' War, it was besieged several times. In 1642, it was taken over and captured by the Swedish commander Torstenson. The castle was then strengthened. For 8 years it served the Swedes as a starting point for Silesia.
In the years 1655-1657, the then owner of Żmigród, Melchior Hatzfeld, extended the castle. In 1691, fortifications were erected. After numerous reconstructions in the second half of the 18th century, it became a baroque magnate residence.