Don't know if any of this helps. From findagrave looks like James remarried to Mary Jane Richey/Ritchey. Didn't find any 1880 census for this family. James Sr. died 1899.
Citing this Record: "Kentucky Marriages, 1785-1979," index, FamilySearch (https://familysearch.org/pal:/MM9.1.1/F491-NXM), James F. Horton and Merica Severs, 1861; citing Union, Kentucky, reference 56; FHL microfilm 562,174.
Name: James F. Horton
Birth Date: 1838 Age: 23
Spouse: Merica Severs
Birth Date: 1844 Spouse's Age: 17
Marriage: 1861 Event Place: Union, Kentucky
Reference ID: 56
U S Census, 1870 Mt Vernon, Posey, IN
Citing this Record: "United States Census, 1870," index and images, FamilySearch (https://familysearch.org/pal:/MM9.1.1/MX6R-CM7), James F Horton in household of Rebecca Mcfadin, Indiana, United States; citing p. 5, family 39, NARA microfilm publication M593 (Washington D.C.: National Archives and Records Administration, n.d.); FHL microfilm 545,851.
Rebecca Mcfadin F 70 Tennessee
James F Horton M 32 Kentucky
America E Horton F 25 Kentucky
George W Horton M 3 Indiana
Susan E Horton F 1 Indiana
Mary A Horton F 0 Indiana
James F. Horton Birth: Jun., 1839 Death: Nov. 10, 1899 Find A Grave Memorial# 53121405
Daughter: Alice Merta Horton Law daughter of James Horton and Mary Ritchey.
Citing this Record: "Ohio, Marriages, 1800-1958," index, FamilySearch (https://familysearch.org/pal:/MM9.1.1/XDJ1-YXH), James F. Horton and Mary J. Richey, 10 Dec 1872; citing Noble,Ohio, reference ; FHL microfilm 0930093 V. 4-5.
Name: James F. Horton
Marriage: Event Date: 10 Dec 1872 Noble,Ohio
Spouse: Mary J. Richey