Question: Is there anything else you know about him? I see in the 1880 Census that he was a carpenter. That might be a small clue.
Also, both children were born in Arkansas, one 3 years ago and the other 6 years ago. So there's a possibility that an E W or E Waldo Kelly might show up in Arkansas in the 1870 Census. There might also be birth records for the two from the Arkansas birth records that could help.
The last name doesn't match, but this record is of interest, because Margaret's middle initial is "M" and it's from Arkansas:
"United States General Index to Pension Files, 1861-1934", database with images,FamilySearch (https://familysearch.org/ark:/61903/1:1:KDB3-V9M : accessed 6 May 2016), Waldo Kelley, 1893.
And likewise, here's a Waldo and Margaret, who had a child with the initials AP (like in the 1880 Census, who would have been born approx 1874) named Albert P Kelley who died in 1953. Her last name is Margaret Aaron here. Or possibly Aaron is her middle name? Or they got it wrong - it happens. But this is definitely his family.
"Texas Deaths, 1890-1976," database with images, FamilySearch(https://familysearch.org/ark:/61903/1:1:K37Q-YRH : accessed 6 May 2016), Waldo Kelley in entry for Albert P Kelley, 23 Jul 1953; citing certificate number 36213, State Registrar Office, Austin; FHL microfilm 2,113,927.