OK, here he is in 1851 in Hove.
James 42, wife Harriet 41 born Houghton, Sussex. James 14, Alfred 12, William 11, Harried 9, Louisa 6 (all born Portsmouth), Emma 2, Ellen 4 months (both born Brighton).
In 1891 shortly before his death he was living with his daughter Harriet (Burns) in Portsea.
That should give you plenty to look for as descendants of those children.