Of course I may have misunderstood the whole problem, so let's check my assumptions.
- You have your Dad, but he's related to everybody, so probably not useful here for this.
- You have a group of 2 people, call them Group A, and they are the only ones currently connected to ancestor A.
- You have the bigger group of 5, call them Group B (B for big), and they are only connected to ancestor B.
What you are testing is the relationship between each individual in Group A against each person in Group B. So you propose ancestors A and B are brothers, then work out what relationship Group A person 1 has with Group B person 1, then check the table to see what the expected total cM should be, and compare that with their actual GEDmatch comparison total. Then propose A and B are cousins, figure out what the relationship would be for the 2 persons again, and check the total for that relationship. Then you can determine which one was a better fit. Then repeat for person A1 with person B2 then B3, B4, and B5, then person A2 with person B1 etc.
Remember that those low to high ranges are not linear, they aren't an even distribution from low to high. They are more like bell curves, with most totals near the average, most probably within the middle third of the range. You should be able to tell if a total is a good fit, or not a fit at all, or a poor fit, near one end of the range.
Have I explained it any better?