The first castle in Elverdinge was built around the 12th century. It originally belonged to the bastards of the counts of Flanders. Adrianus Vander Borcht received the manor in 1629. He had a castle built in Flemish Renaissance style. It was surrounded by water. This castle stood on the site where the old monastery now stands.
After the Vander Borcht family, the manor came into the hands of various families. In chronological order the families: de Bever, de Lichtervelde, de Steenhuyse, de Bethune Hesdigneul, d'Ennetières d'Hust and de Laubespin.
Marquis Victor d'Ennetières d'Hust (1826-1908) settled in Elverdinge in 1870. He expanded his domain by purchasing land. Thanks to him, Elverdinge became a significant municipality. He turned the park into one of the most beautiful parks. In 1870 he had a new castle built in French Renaissance style and a family crypt. He had all the streets (14 km) laid out in macadam at his expense. Because of his mildness he was called the "father of the poor". His only daughter married the Frenchman Alfred Mouchet Battefort de Laubespin in 1872.
The castle burned down during the First World War, but the reconstruction of both castles began in 1925. The "new" castle was rebuilt from the existing walls. The only notable change is the absence of a torelleke and a simplified roof. The "old" castle was rebuilt in local yellow bricks with a French Mansard roof, in the Louis XV style.
During the Second World War, German staff officers moved into the castle, so it had to be renovated again after 1945.
The castle is still owned by the descendants of the de Laubespin family: Count Eric du Cauzé de Nazelle and his wife Isabelle de Rochechouart de Mortemart.
The tree-rich park has been a free bird sanctuary since 1957 and is supervised by the Belgian nature and bird reserves. The castle cannot be visited, but it can be seen from a distance.