The history of Gram Castle can be traced back to 1231 when it belonged to King Valdemar Sejr. The oldest part of today's three-wing structure is the east wing, which was built in the 16th century.
The south wing was added in 1670 and the west wing in 1751. The builder of the south wing was Hans Schack - one of the richest and most powerful men of the time. Hans Schack, who had made a career as major general of the French army and commander in Hamburg, was contacted in 1657 by the Danish government, who offered him a prominent post in the Danish army, which was at war with the Swedes.
Schack knew his own worth and demanded, among other things, a Danish nobility title in order to sign the agreement. He received the title of liege and was appointed lieutenant general. As a reward for his efforts to defeat the Swedish King Karl 10. Gustav Hans Schack was appointed Imperial Warlord and handed over to Møgeltønderhus in South Jutland, which later changed its name to Schackenborg. In 1664 Hans Schack bought Gram Castle, where he built a new breeding business, including the Holstein barn.