The castle probably dates from the 10th or 11th century and was originally owned by robber barons who obtained income by charging tolls on traffic on the busy Meuse. The oldest mentioned ruler of the area around the village of Horn is Engelbert van Horn, son of Dirk van Horn. The castle is first mentioned in a letter from 1243. It states that William, Lord of Horn and Altena, has borrowed the castle and the nearby village from the Count of Loon.
Because the Maas changed its course, the castle came to lie so far from the Maas that the possibility to levy tolls disappeared. Willem I van Horn left the castle and settled in Die Haeghe, later Aldenborgh, in Weert.
When Jacob I van Horn received his count title in 1450, the castle became the center and administrative center of the county of Horne, an independent mini-state. The last count in the Van Horne dynasty was Philip of Montmorency. He was beheaded by Philip II in 1568, together with Count Lamoraal van Egmont on the Grand Place of Brussels, which would be the starting signal for the Dutch Revolt against the Spanish rulers of the Netherlands.
Castle Horn has been inhabited by a steward since the middle of the thirteenth century because the Van Horne family usually stayed in Castle Nijenborgh in Weert. In 1798 the castle was sold to Marcel-Gérard Magnée from Liège. The Magnée family still owns it.
In 1948, during plumbing work, a large fire broke out in the living area. Between 1954 and 1957, the main hall was restored to what it was in the 15th century. The castle is still inhabited by the Magnée family, who also call themselves Magnée de Horn because of their property.