James was living with his brother at age 63 ("single") during 1910 Fluvanna County, Virginia Census
Note sure why James was not listed with the family in the 1850 Richmond, VA Census since he would only have been 3 years old.
Have you taken a DNA test? If so, login to add it. If not, see our friends at Ancestry DNA.