I have one ancestor who's family got completely split up as he, his wife, his sons and youngest unmarried daughter all left for Nova Scotia, but his older married daughters stayed in MA with their husbands and kids. The letters sent 10-20 years later are truly heartbreaking as it's clear that they rarely if ever got to see each other again.
As the comments below indicate, only those who were accused and "convicted" as loyalists are officially listed as such, the rest of the family is just collateral damage to the decisions.
On another note, I have a Scottish ancestor who was captured and shipped off to the colonies during the Jacobite rebellion of 1715 and there's no further mention of the wife he left behind. His baby daughter grew up in Scotland and got married and had her own family, but he never returned and it's my firm belief that they never saw/heard from each other again. In fact, all we know is that he remarried in MD and had an entirely different family from which I descend.
Being on the losing side of a rebellion/war definitely has negative consequences for the non-combatants, too. It's not just the soldier who suffers.