You can see evidence of his father, mother, and siblings’ identity on the 1830 census. As was common back then, he married a neighbor’s daughter, Margaret Isbell, in 1847 when he was 20 years old. The Isbell’s lived close to Jeremiah Willis according to the 1830 census. There are other records of a Jeremiah Willis and Sarah Davis that died before the 1840 census...just like James parents. The ages of the other household members and siblings match across the censuses.
To me the evidence points to his parents being Jeremiah Willis from KY and Sarah Davis. The 1820 Breckinridge KY census has a Jerediah Willis with a similarly matching household.
I hope this helps.
Kim Britt