I'm happy to find that someone else has a misalignment hypothesis.
Although, from what I can tell, if Black laborers lived on a farm in New England pre-1850, Black people would be enumerated, but only the farmer's name would be recorded.
So, like you, I do consider the idea that the enumeration may be misaligned, however, who the heck knows?