View of Chapel Ruins at Chapel Ness

Earlsferry, the older of the two villages, was first settled in time immemorial. It is said that MacDuff, the Earl of Fife, crossed the Forth here in 1054 while fleeing from King Macbeth. In particular the legend tells of his escape being aided by local fishermen, an act which may have led directly to the village being promoted to royal burgh status due to MacDuff's later influence over Malcolm III.

By the middle of the 12th century, the Earls of Fife had instituted a ferry for the use of pilgrims en route to the shrine of Saint Andrew the Apostle at St Andrews. The ferry crossed the Firth of Forth to North Berwick, a distance of 7 miles, and it is this ferry that led to the naming of the place. There are the remains of a small chapel on Chapel Ness, built for the use of these pilgrims.