How sure of the birth date (and other details) are you? There are several John Thayers born at about that time:
John G Thayer, age 58, is in the 1850 Census at Evans, Erie County, New York. I guess this is your ancestor. Other Thayers in the household are: Mary age 31, George A age 7, Prudence M age 4, John P age 3, and Caroline age 21.