The first mention of the castle dates back to the second half of the 10th century. In a deed of gift from 964, Arnulf I, the great, the Count of Flanders, donated part of his lands to the St. Peter's Abbey of Ghent, including the Castle of "Potegem"[source?] next to Desselgem, Beveren-Leie and a part of Waregem. Somewhere around 1260-1270, the castle was taken from the possessions of the abbey and formed a lordship.
The house then belonged to the Lords of Potegem who controlled the lordship of Potegem, part of the castellany of Kortrijk. Later, at the beginning of the 14th century, the Treaty of Athis-sur-Orge split the castellany of Kortrijk into a castellany of Kortrijk and a castellany of Oudenaarde. From then on, the lordship of Potegem belonged to the castellany of Oudenaarde. The Gaverbeek was the separation between the two and so the old Waregem was split through its centre. The baron of Ingelmunster was the Lord of Waregem, the centre with church, the lord of Potegem on the other side. However, the lord of Waregem had jurisdiction in both low, middle and high degrees and therefore had more influence.
In the 16th century, Walter van Potegem, third count of the lordship of Ingelmunster, was a resident of the castle. Due to his participation in the resistance against the Spanish rule, the castle was partly burned down by Fernando Álvarez de Toledo in retaliation.
Also in the Flandria Illustrata of Antonius Sanderus in 1641 it is described as a spacious, beautiful mansion. The castle of Potegem was already mentioned in the second half of the 10th century. With the abolition of the ancien régime in 1795, the lordship of Potegem became part of the municipality of Waregem.
The castle had lost its function and became a boys' boarding school in 1830, where Felix de Ruyck was also educated. This was founded by Ivo Van Robaeys and after him it became the property of the OCMW.
After a few years of decay it changed owners again. The industrialist Robert Vindevogel bought it from the OCMW in 1930 and had it restored to its current appearance. The architect Viérin gave it a neo-baroque style.
The castle then came into the possession of the Devos family.