The precise date of construction is unknown but it is generally thought that the original tower was built around 1765 to 1770 by John Hanbury, a local landowner and ironmaster who owned Pontypool Park. The local name of The Folly was in use as far back as 1865 when the Free Press of Monmouthshire described it as:
"...an elevated spot where a Tower (formerly a Roman watch-tower) was many years since rebuilt as an observatory and which is popularly known as 'The Folly'."
It is thought the tower was renovated around 1831 by Capel Hanbury Leigh. This date is derived from a keystone from the doorway bearing an '1831' inscription. The tower provided a vantage point over the local area and quickly became popular and a well known feature of the area.
In May 1935 an estimated 15 to 20,000 people gathered at the Folly Tower to celebrate the Silver Jubilee of King George V with the lighting of a bonfire.
However, the folly, being open to the elements, began to fall into a state of disrepair during the late 1930s. Pontypool Park Estate Office attached notices to the structure informing visitors of the potential dangers of falling masonry. Its draw on the locals though remained unstinted and local poet and illustrator Myfanwy Haycock penned the following verse in 1937: