The Castle of Seneffe was built in the period 1763-1768 in classicist style according to the plans of architect Laurent Benoit Dewez, as a country residence for the wealthy entrepreneur Julien de Pestre, Count of Seneffe. His son Joseph Depestre inherited the castle in 1774. He had the garden laid out in consultation with his mother Isabelle Cogels. There were statues by Augustin Pajou, a theater by Charles De Wailly and an orangery by Louis Montoyer. The castle and the art collection were sold as national property in 1798, but Joseph's brother Jean-Baptiste managed to keep it in the family through a stooge. His son Honoré de Pestre was able to move in in 1823 and sold it in 1837 out of financial distress.
During World War II, the castle served as the local headquarters and summer residence of General Von Falkenhausen, the military governor of Nazi-occupied Belgium. The castle was neglected afterwards. After changing hands many times, it came into the hands of the Ministry of the French Community of Belgium, which decided to house a museum of silversmithing.
The castle underwent a lengthy restoration, which was completed in 1995. In 2008, the extensive gardens surrounding the castle were recreated, partly according to the original plans.