This castle was built, together with the walls of the Upper Town, by order of Philip Hurepel, Count of Boulogne and son of Philip II of France, from 1227-1231. There have been various adjustments over the centuries. Major renovations were carried out by the Duke of Berry between 1394 and 1416. The horseshoe shape (barracks, arsenal) was completed around 1567. The towers date from the 16th century and were partly built after the English destroyed part of the castle. After adjustments due to developments in artillery, it lost part of its medieval character in the 16th century.
From the 17th century the castle was used as a barracks and after the Second World War as a prison. In 1974 the castle was purchased by the municipality. The Castle Museum was opened in 1988. The castle has a polygonal plan and also includes part of the fortress walls. It has nine round towers.