I actually suspect that, in some cases, family members just pulled a date out of thin air or made a guesstimate. I have definitely seen cases where, for example, "Jane" is 5 years old on the 1860 census, but her tombstone claims she was born in 1865. Clearly, "Jane" shaved a few years off her age at some point and her descendants went along with it.
For what it's worth, I tend to trust the ages estimated for very young children rather than for adults. A census-taker is less likely to confuse a five-year-old for an eleven-year-old than a 25-year-old adult for a 30-year-old adult.