When William Drake bought the Manor of Amersham in 1637, his surveyor mapped a large area of arable and woodland to the south west of Larkins Green. The three fields fronting the Beaconsfield Road, White Hill Pightle, Broad Field and Long Field, were leased in 1638 to William Ball of Amersham, draper. The remainder, including a field called Horse Lease, was sold separately. A house was later built on Horse Lease by the Quaker, Ralph Trumper. Some of the early Quaker meetings were held at Trumper's house, and the famous Quaker, Thomas Ellwood, came to live nearby in 1669. It is not clear why Trumper and Ellwood chose to call their new home Hunger Hill as it was not the name of a locality or of one of the fields. The Quakers faced fierce persecution and may have chosen this out of the way location in the Hertfordshire part of Amersham knowing that the Buckinghamshire J.P.s had no jurisdiction there.

Hunger Hill was later mortgaged to another Quaker, Henry Child of Cold Harbour (the house near the present-day Water Tower). When Child sold it in 1690 to Daniel Wharley of London, woollen draper, for £1200, it was called Chappell Farm, possibly a reference to the meeting house:

All those two messuages or tenements with the appurtenances being heretofore one messuage and then called or known by the name of Chappell Farm heretofore erected and built upon part or parcel of a close of land lying and being in Coleshill aforesaid in parish of Amersham Co Herts called Horse Lease and in which said messuages one Ralph Trumper and Thomas Ellwood gent do now severally inhabit and dwell

Ralph Trumper died in 1692, leaving £15 for 'the relief and support of such of my poor friends called Quakers in parish of Amersham'. Ellwood continued to live at Hunger Hill until his death in 1713.