The construction of this castle began in 1391. It was inhabited by the Knights of the Order of Germany and it was commissioned by the Commandant Hendrik Reinaart van Husen. It was located on the Rips south of the village chapel. This chapel was to the east of the current St. John's Church from 1437, on the other side of the street.
In 1645 the castle was described by Philip van Leefdaal as a magnific out ghebout castle with dry scoring bridges and the two beautiful well-timed down courtyards, residing in the residence of the commanders and belonging to the commandeuryen with other beautiful goods, as being Gemert the ryckxste commandeurie under the baliage of Oldenbisen.
In 1648 the local commander tore himself away from the Order, but after lengthy legal proceedings, the authority of the Grand Master of the German Order over Gemert was recognized in 1662.
In 1740 a new main building was built in Louis XIV style consisting of three wings surrounding a courtyard.
In 1794 the French seized the goods of the German Order, and this was abolished in 1809 by order of Napoleon Bonaparte. The castle came into the hands of Marshal Nicolas Charles Oudinot. It was later sold to Adriaan van Riemsdijk and in 1832 he bought a number of farms, windmills and farmlands that had previously belonged to the German Order.
The castle has since been inhabited by a steward, and in the western part there was for some time the cotton mill of the company Volkert & Comp. established, see Gemert textile industry.
A fire broke out in 1881, causing major damage to the western and southern wings. This happened just after the castle was sold to the Jesuits. These were expelled from France in the context of the Secularization policy there. They established their novitiate there in 1900, but they went back to France when the First World War broke out.