The first predecessor of the present-day castle was built shortly before 1340 as an allodial property of the von Neuroth family of castle knights. In 1584, after several changes of ownership, it passed by inheritance to the lower noble family of Brambach. This united the associated estates with those of an older ministerial family of Neuroth, who had already established a farmstead elsewhere in the district before 1222. Furthermore, the House of Brambach acquired other scattered rights to the small settlement, thus creating a unified estate.
In 1687, Johann Philipp von Brambach sold the castle for 13,000 Rhenish guilders to Wilhelm Rheinhard Freiherr von Walderdorff. The family, elevated to the rank of Imperial Lords in 1660, was pursuing a strategy at this time of consolidating its holdings around its ancestral seat, Molsberg Castle. The acquisition of the Neuroth estate likely falls within this context.
Since the time of the Brambach family, and from the second half of the 19th century onward under the Walderdorff family, the estate served as a secondary residence for the family. When no family members resided there, it housed foresters and administrators of the count's estates. The tenants of the adjacent farmyard were responsible for the upkeep and maintenance of the manor house.
The Walderdorff family still owns the estate today. They continue to use it as their residence and farm numerous fields in and around Bilkheim.