Designed by James Gillespie Graham (1776–1855) in 1813 after original plans by Robert Reid in 1807. The original plans had called for a courthouse and three prisons, one for males, one for females and one for debtors. The ground obtained was sufficient for such an ambitious plan, but the finance was not and the Prison Commissioners had to be content with only one prison.
Both the courthouse and prisons opened in 1820. The courtroom, on the first floor, has a semicircle of large windows giving a magnificent view overlooking the prison yard and, beyond, across Loch Fyne.
The two-storied prison has three-foot-thick walls of massive rough hewn red stone and originally contained cells on both floors, eight in total. A third of the ground floor was occupied by a day-room which was lit, like the cells, by narrow, unglazed windows. The Prisons (Scotland) Act 1839 brought about many changes, including the separation of prisoners. A second prison was finally built on the spare ground, opening in the closing days of 1848. Designed by Thomas Brown of Edinburgh, the new prison consisted of twelve cells on three floors with an exercising gallery at the top. A pair of outdoor exercise yards were also built, separated from each other by a wall. Prisoners were not allowed to fraternize. The separate system was designed to give them time to reflect upon their sins.
The Prisons (Scotland) Act 1877 heralded another major change in the administration of prisons. Local authorities would no longer have the responsibility of running and financing prisons; the Government was taking control. Large prisons were built in the center of population. Barlinnie in Glasgow opened in 1882 and small local prisons in the West of Scotland began to close. Inveraray would be the last. On 20 July 1889 readers of the Argyllshire Herald were informed that "the prison of Inveraray will be closed on 31st August".
Despite changes to the prison system, the court was still sitting, continuing until 1954, with a little activity since then. On several occasions the premises have come near to being abandoned, especially when it was necessary to spend money on repairs. In the 1980s extensive restoration was undertaken by the Scottish Office, but then all plans fell through and the building lay empty.
In May 1989, Inveraray Jail was reopened as a privately owned tourist attraction, re-enacting the trials and the life of the inmates of the 19th century.