I have just added Census information and some notes to James' profile.
I found a possible Census record in 1860 for James, age 10, and his mother, Martha.
The household immediately previous to her is a large Willoughby family, possibly hers, although this would mean she was unmarried at the time of his birth... just a possibility so have a look and see what you think. I wish there were more records for Kentucky for you!!