Yes, I know. Census records are a real pain. Especially the pre-1850 ones.
However, there was only one Griffin family in Lower Regiment, Chatham County in the 1850 Census, and the members of that family were Sarah (mother), John, Anne, Abel, Catherine, Jane, and Alexander. (The latter three are spelled Griffon instead of Griffin, but they're listed in the same household - go figure.) John's father died before the Census, and those are his siblings and mother, so that has to be him. He's listed as a laborer and he's 21 years old. (Ages are kind of squishy in the Census records.)
So, that's definitely him, and his entire family.
In 1860, the Upper and Lower Regiment designations were gone, and it was now Eastern, Middle, and Western Division. So that makes it a little harder to figure out. But, in the Middle Division, there are four Griffons listed: John, Mary, Sarah, and Alexander. Mary, Sarah, and Alexander were the three youngest of that family, so it appears that John was now caring for his siblings, who were teenagers by this time.
It isn't until 1870 that Julia shows up, but suddenly there' are three older children who were born in 1859, 1861, and 1862, along with John's (what I think are) first two children with Julia.
So, the only explanation I can come up with for the three older children is that they were Richard Ausley's.
They didn't have birth certificates in the 1800s, and the marriage records are basically bonds that some people ignored, because of the cost. So, finding birth certificates is a non-starter, and finding marriage evidence is spotty at best.