I think James Millard's wifes name was Harriett Millard. You can see how this confusion could have arisen through bad writing on documentation.
In the 1841 census you will find them living at Cross, Compton Bishop, Somerset.
They have a 4 year old son. Their age estimates are James 35, and Harriett 32. James is an agricultural labourer. Harriett is at home.
In 1851 tey are still in Cross, James is working as a Groom and is keeper of The Kings Arms Tap. Harry is not at home as he is now 14 and is probably working away on another farm at the time of the census.
The 1851 census says that James was born in 1805, in Compton Bishop, Somerset . Harriett is now stated by the enumerator to have been born in Burrington, Somerset, in 1804. I think he may have made a transcription error here again with bad writing of 1809.
I was unable to locate them in 1861 but in 1871 census they are still living in Cross but it looks as if Harriett is telling fibs about her age as this is given as 55. In this census her birthplace is given as Keynsham, Gloucestershire.
It seems very unlikely that there could be two families of James and Harriett Millard living in Cross so you will need to sort out the truth about Harriett and see if Harry links down to your tree.
I was unable to find any record of a marriage and theirs may simply have been an agreed marriage between themselves.
There is some information on Cross at :
Good luck with your researches