Hans Jakob Khönle (1618-1675), son of a miller from Hildrizhausen, learned the trade of a merchant in Frankfurt or Hanau. In 1651 he married a baker's daughter from Herrenberg. At first he traded in iron goods, which were in high demand during the reconstruction period after the Thirty Years' War. His business flourished and he expanded the range of goods. The outward sign of his success is the increase in his real estate holdings. As early as 1652, Hans Jakob Khönle bought a farmstead on the market (Marktplatz 7, former market pharmacy), on which he built his first house. He also bought the farmstead next door (Marktplatz 6, built in 1663/64), an undeveloped farmstead on Tübinger Straße and in 1667 the house at Schuhgasse 2.
His son Johannes Jakob Khönle (1652-1731) continued the company with great success. He had four shops in Frankfurt, and others in Tübingen, Horb and on the market square in Herrenberg. Through his connections in Cologne and the Netherlands, he also sold overseas products, so that in the Khönle shop in Herrenberg, in addition to metal commodities and a wide variety of European products, one could also buy tobacco from Guyana, tin from India, incense and spices from the Orient, Turkish strings, porcelain from China and coffee from Java. In 1678, Johann Jakob Khönle acquired the Hofstattplatz on the corner of the market square and Tübingen street, so that the Khönle family owned almost the entire south side of the market square. He also bought and managed the third Ammermühle.
The carved corner pillars, which depict Johann Jakob Khönle and two of his four wives as embodiments of the seasons and which are on display in the fruit box, come from this building. Thanks to his wealth, Johann Jakob Khönle was able to help the town of Herrenberg financially when it was threatened by French troops during the War of the Palatinate Succession and when the military repeatedly camped in the town. His third marriage produced his son of the same name, Johann Jakob (1689-1733), also a successful merchant who opened his own shop in 1717 on the corner of Marktplatz and Tübingen Street. His widow married Christoph Adam Erhardt from Urach (1704-1773). He successfully continued the Khönle family business and was also mayor of Herrenberg from 1752-1766. Since he and his wife had no descendants, he invested his fortune in charitable foundations and donations for church purposes.