The building was erected as a defense structure at the entrance to the Oude Haven, and was originally located on the West Frisian Omringdijk; this ran from the north via the Breedstraat further along the Zuiderdijk. The section of dike near the Drommedaris was later excavated for the construction of the Buitenhaven. The gun cellar and ground floor space date from 1540 and were the original structure. Both rooms contained cannon holes along the walls in niches where cannons were located that could cover the harbors. The upper space has a ribbed vault. Above the building on the roof was a structure containing prison cells that can still be found on the first floor of the current building. Above the gate is a cell for death row inmates where dates and sometimes poems are carved into the oak paneling. Outside above the entrance to this gate, the Emperor's coat of arms was carved in white stone with the text below: "Blessed is the City and highly praised, which demands war in time of peace." A bell hung in a wooden bell tower on the roof to announce the closing of the gate.
The old name of the building is Zuiderpoort or Ketenpoort and later also Wilgenburg (for the stronghold to which the tower belonged). The name Ketenpoort refers to the salt chain south of the city along the Zuiderdijk, which could be reached via this gate. Here seawater was evaporated to obtain the salt with which herring was preserved. In the 19th century the name Drommedaris or Domburg was used.