They market these DNA tests as a sort of "instant genealogy", in terms of determining the ethnic mix of your ancestors, but that aspect of the results has little genealogical value. All it's really telling you for sure is that you're of European origin - something you plainly see in the mirror.
People really WANT to believe the ethnicity is accurate, and it's a revealing exercise in psychology, but not genealogy. You'll see people with completely goofball ethnic mixes (although generally from the right continent), explain how it's really OK - usually involving discussions of Roman or Norse invasions - really trying to rationalize something that's really just junk. It's DNA, it comes from a lab … so it HAS to be right, doesn't it? Well, no it just doesn't.
In contrast, your DNA matches are very real, and genealogically useful. There are some "curve balls" it can throw you, but THAT'S the stuff to pay attention to.
It sounds like you may not have very many useful matches, which is a shame, but perhaps not very surprising. This is a bigger deal in the US (apparently people in France are already well aware that they're French, so it's like that).
Not to sound like an AncestryDNA salesman, but they have something like half the tests ever done in the world, so if you really want more matches testing there would be the way to go. Beyond that, get your results up on GEDmatch.com (a free site that allows people using different testing companies to compare results).
Matches down at around the 60cM and below level can be anything from a 2nd cousin, once removed to an 8th cousin, so it's probably an uphill battle to try to do too much with those, at the beginning.
It doesn't sound like you have the slightest reason to suspect any adoptions, or illegitimate births, so far, based on what you're saying.
By way of example, my own AncestryDNA result said I was 78% "Great Britain", even though my mother was 0%. My mix didn't even match those of my two brothers. YET, I can tell with certainty that they are both full brothers, from the cM of my matches to them, and that my mother was my biological mother from the large number of matches I have on her side, including 3 first cousins (who also have appropriate levels of cM). My tree is almost entirely DNA confirmed back to my gt-gt grandparents, and on my mother's side that means "back to the Old Country". So there's no doubt at all - the 78% was just nonsense.
Further, my 90% Great Britain and Ireland (78%+12%) has since been revised to 47% "Ireland and Scotland" + 43% "England, Wales & Northwestern Europe". My mother was half Irish, and half from German-speaking places, so that can be interpreted to be somewhat reasonable. Still, note that since the two estimate are DIFFERENT, at least one was obviously WRONG.