You are absolutely correct!
However, I have been using the USGS maps (U.S. Dept. Geological Survey) a lot lately and found them invaluable in my research. For example, 150 years ago there were "clusters" of my family in various parts of the country. I can order maps for a given date and area (almost all are FREE to download, but huge files) which go to the house level, showing many things in the surrounding area as points of reference. So occupations, plots of land, railroads (many no longer in existence), factories, schools, etc. are on these maps. They give me a context to further inquire exactly where various relatives were located (coinciding with census and other data) and then allow me to check local land title, probate court records, university and other archives, etc.
It helps me to fill in some of the "hard to collect" information, such as common law marriages. So far I know of no common law marriages for my direct ancestors, but I'm sure they can be used if needed.
Canada and other countries also recognize common law marriages.
Agreed, not the easiest way to go about it! But just a suggestion for some "brick wall" situations.