Traditionally, this was an open field in the middle of the city, which belonged to the Onze-Lieve-Vrouwekapittel in Tournai. It was therefore called den Canunckenbosch. Part of the land was at one point donated to the Onze-Lieve-Vrouwekapittel in Kortrijk. The property was confiscated by the French and sold publicly.
At the beginning of the 19th century, it was bought by Godefroid Boulez, who was mayor of Waregem from 1804 to 1819. Around 1816, he had a park laid out and several buildings erected within an 8-shaped moat.
Godefroid's daughter Victoire was married to Ferdinand Storme, who was mayor from 1819 to 1830. She inherited the property in 1829. Their daughter Nathalie married brick manufacturer Felix de Ruyck. They commissioned the construction of the castle and also expanded the domain. In 1852 they moved in.
The park was laid out in English landscape style, and the moat was dug out to create an ornamental pond. A castle with a rectangular plan was created. In 1872 the domain was expanded and in 1878 the castle was also given an extension. In 1883 a portico was built and a new coach house was also added.
In 1897 the domain came to Baron Victor Casier and his wife Marie-Victorine Storme. They used the castle as a summer residence. After 1911 their daughter, Agnes Casier, had a chapel built.
During the First World War the castle was badly damaged and the family archive was also destroyed by fire. In 1919 the castle was restored to its original state. The gardener's house and the pavilion, which had been built in 1816, were demolished.
In 1977 the park was purchased by the municipality. After thorough restoration of the castle and park, the latter was opened to the public as Park Baron Casier from the early 1980s. In 1983, catering facilities were added to the castle.
In the 1990s, a connecting road between the Market and the park was constructed.