The economic decline of the von Stein zu Altenstein family forced the family to abandon its nearby ancestral castle of Altenstein at the beginning of the 18th century. As a new residence, the von Stein family had the small castle built in Pfaffendorf in 1703. The new building replaced an older residence from the first half of the 16th century.
The planning of the Pfaffendorf castle was the responsibility of the Franconian master builder family Dientzenhofer, and the construction was carried out by the engineer lieutenant Michael Küchel, a student of Balthasar Neumann. The builder Christian Adam Ludwig von Stein had the building completed in 1763. The three-wing, two-storey complex appears simple on the outside and impresses with its clear architectural structure.
The English ladies acquired the property during the Great Depression. After the Second World War, the Salesians of Don Bosco built a school and boarding school there (Pfaffendorf Youth Welfare Center), which is specifically dedicated to supporting children and young people with learning disabilities and "educational difficulties".