Raudonė is first mentioned in the 16th century as the estate of the Polish King Sigismund II Augustus. He sold the property around 1580 to the Prussian timber merchant Hieronymus Krispin-Kirschenstein. He and his son built the first palace in the Renaissance style.
The palace owes its present appearance primarily to the Neo-Gothic reconstructions carried out between 1861 and 1877 under the direction of Count Zubov's daughter, Sofia Kaissarowa. The last noble owners were the granddaughter Sofia Waxell and her Portuguese husband José Carlos de Faria e Castro.
Following looting during the First World War, the de Faria e Castro descendant was in debt and auctioned the palace in 1934. Following protests, the Lithuanian National Bank ceded it to the Ministry of Education, which undertook restoration work in 1936. The Wehrmacht blew up the castle's main tower in 1944, but it was later rebuilt and is now open to visitors. The main building has housed a middle school since 1947, which was downgraded to an elementary school in 2003.