You appear to have the right idea.
If you look at the charts on https://dna-explained.com/2012/09/27/x-marks-the-spot/
you will see that boys can only inherit their X-chromosome from their mother while girls get one from each parent so because your nephew is descended form your sister he gets a lot that is the same as you. If he was descended from your brother he would have zero amount of X in common with you. All those relatives who have any X in common with you are descended from you via grandmothers, ggrandmothers etc