Well - maybe one other possibility...
I looked into William Seth (his proposed father). Looks like the family may have moved to California, where his mother likely died.
"California, San Francisco Area Funeral Home Records, 1835-1979," database with images, FamilySearch (https://familysearch.org/ark:/61903/1:1:JNT2-CP5 : 5 August 2017), Ellen J Seth, 09 Nov 1858; citing funeral home N. Gray & Co., San Francisco, San Francisco, California, record book p. 1-335, 1854-1861, San Francisco Public Library, San Francisco History and Archive Center.
His father appears to have remarried and had several other children:
"United States Census, 1870", database with images, FamilySearch (https://familysearch.org/ark:/61903/1:1:MN65-HM5 : 19 March 2020), William Seth, 1870.
"United States Census, 1880," database with images, FamilySearch (https://www.familysearch.org/ark:/61903/1:1:M6PH-XQL : 12 November 2020), William H. Seth, San Francisco, San Francisco, California, United States; citing enumeration district ED 59, sheet 752A, NARA microfilm publication T9 (Washington, D.C.: National Archives and Records Administration, n.d.), FHL microfilm 1,254,074.
If this is his family, it would be interesting to find out why they left little Joseph behind when they traveled to San Francisco, and put him in the hands of his Aunts/Uncles of the Bowers family. So far all I've got is speculation but it would be nice to find something concrete to lean against.