The 1841 census has him living in Falmer, Sussex, and the 1851 census has him living in the same place and working as a road labourer.
This information was provided by Jane (Stoneham) Payne, a direct descendant, available through a paid Ancestry.UK account. The following list of sources are used on the Ancentry account but are otherwise unavailable to display further here:
1841 England Census; Registration district Lewes; Sub-registration district Rottingdean; Piece 1113; Book 11; Folio 6; Page number 5.
1851 England Census; Registration district Lewes; Sub-registration district Ditchling; ED, institution, or vessel 5; Household schedule number 16; Piece 1643; Folio 817; Page number 5.
These two sources imply the same family because of similar names, locations, and dates, however, half the family seems to be missing in the 1851 census, and different birth years are implied for all family members.