The origins of the castle date back to the tenth century. Around 980, Heribert, the bishop of Auxerre, built a fortified hunting lodge.
At the beginning of the 13th century, Jean de Toucy, a close confidant of Louis the Saint, built a pentagonal castle made of iron sandstone.
His daughter married Thibaut de Bar in 1255. The House of Bar ruled the Puisaye countryside until the end of the Hundred Years' War, which drove them into bankruptcy.
In 1450, the castle and lands became the property of the richest man in France at the time, Jacques Coeur. He was financier to King Charles VII of France. Shortly thereafter, Coeur was arrested and dispossessed by the Crown.
In 1453, Antoine de Chabannes, a former soldier and close confidant of Joan of Arc, bought the entire property. As an enthusiastic military expert, he converted the first castle into a strong fortress. He covered the walls with a thick covering of bricks and built 6 massive and contemporary fortified towers at all corners.