Not an answer but another lead -- in 1700's in New Hampshire at least one Willoughby family was active. In Grafton County, there was a "Willoughby Hill".
In the 1950's, Al Willoughby had a dairy farm and owned land, including a pond, in Bloomfield, Hartford County, Connecticut
Perhaps it's time for a Willoughby one name study?