In the 13th century, there was a castellan stronghold here, which served as a border stronghold. After the castellany was liquidated in the second half of the 13th century, the castle passed into private hands, becoming the property of the Świnka family, who gave the castle its name. In the 14th century, the castle was rebuilt into a brick fortress, and in the 15th century it was expanded by Gunczel von Schweinchen, who built a new wing of the castle. In 1620, after the Thirty Years' War, the castle underwent a major modernization. Jan Zygmunt Świnka built new walls, bastions, and a drawbridge, which was to strengthen the castle's fortification function. In his time, Świny became one of the most powerful seats in the region. However, after Jan Zygmunt's death in the second half of the 18th century, the castle fell into ruin.
In the 20th century, Świny Castle passed into the hands of various owners, who attempted to renovate it. In 1905, the building was secured by the Association of Friends of Bolkowy Gaj, and later gradually opened to tourists. Unfortunately, the castle was damaged during World War II, and after the war it was abandoned. Currently, the castle is being restored, but is open to visitors. Admission only in the summer (from May to October).