Hi James, this is not a complete answer but merely if I may, an educated opinion.
I have found 7 children born to James Mair and Ann Hunter. In your request this morning you hadn't mentioned Ann Mair who was born in 1841 St. Quivox as well.
Ann Hunter is on the 1851 census aged 40 and as you mentioned James may have been a seaman as he is absent the night of census taking and Agnes is noted as wife meaning she was not the head of household.
Elizabeth Gardner Mair is a great clue to search. (Elizabeth Gardner) may be the mother of James or Agnes according to Scottish naming patterns.
The first son born from my search was William and that may be the father of either James Mair or Agnes Hunter.
Names and dates for children that I can find:
Agnes 1832, William 1834, Andrew 1836, John 1837, Elizabeth G. 1840,
Ann 1841, James 1844
Wishing you well in your search.