The building, formerly also called Walensteyn, was mentioned as early as the 13th century. At the end of the 13th or early 14th century, two floors were built, after which the current height of almost 15 meters was reached.
The keep is probably from the 14th century, and stood alone for many years. The tower originally consisted of only one room above which was a battle platform. The facade was then 6.5 meters high. The walls are 120cm thick. The building rests on vaults and corner pillars. A residential wing was added around 1550.
At the end of the 14th century, the tower came into the possession of the De Ridder van Groenenstein family, who owned several homesteads. Nicolaas de Ridder was the owner of the castle until 1574. When he had too many debts he had to sell it to Johan van Lendt in Utrecht. Four years later, he sold it to Johan van der Meer, who sold the Walenburg to Jan van Hattum van Rhijnestein in 1582. That same day, the Walenburg was sold on to Johan Botter van Snellenburch.
In 1648 the castle was sold to Diederik van Doeyenburch à Cuylenburch, who kept the castle until 1661. The next owner was George Johan baron van Weede. De Walenburg remained in that family until it was sold to Jan Robbert Mollerus in 1733. In 1761, the Walenburg was sold to Johannes Mathys Swemmelaar and his wife Anna Maria van Plasburg. After her death, her son is enfeoffed with the estate in 1792.
In 1803 the castle came into the possession of the van Lynden van Sandenburg family. Gijsbert Karel Cornelis Jan baron van Lynden van Sandenburg had the outbuildings demolished and a farm was built against the tower.