A medieval castle dating from the 13th century previously existed on the site. By 1536 it lay in ruins. By 1563 the site had been sold to a Dr John Gibbon. A new house was built on the site either by Gibbon or by Nicholas Herbert, who bought the site from Gibbon in 1586. Part of the D-shaped medieval boundary fortifications remain, forming a wall around the current house.
Sir Edward Lewis of The Van, Caerphilly, bought the house in 1616 and the interior dates partly from then and partly from after 1850, when it became the summer residence of the Windsor-Clive family. The property was part of the estate of the Earls of Plymouth and, in 1833, was inherited by Lady Harriet Clive who proceeded to restore the building. The sequence of terraces in the gardens was created for the Windsor-Clives in 1865–6 and extended in the early 20th century. The house became a convalescent hospital for soldiers during World War I, with the banqueting hall containing a ward of 40 beds.
In 1947 the Windsor-Clives gave the house and grounds to the Amgueddfa Cymru - National Museum Wales and the grounds subsequently became the site of the Welsh Folk Museum, now St Fagans National Museum of History.
St Fagans Castle became a Grade I listed building in 1977. The lead water cistern in the courtyard of the castle is Grade II* listed, while a number of features in the gardens are listed as Grade II.