On his death, have you made a list of his children, not just your direct ancestor, and checked the areas around them? His wife died in 1864, and, in the 1880 census, he was 72, without any family living with him. Given that he's not buried with his wife, it makes one wonder if he was relocated to live with a child.
I have had several ancestors where they didn't die and weren't buried in the area that they had lived for all or most of their lives and, most times, I have found that they moved to live with a child - most often a daughter - and died there,