The 1841 census actually has "N" for born in county which makes sense as James considered himself to have been born over the border in Rutland. His age is given as 30 which is doubtful. I would normally read his occupation as "Tailor" but given his later police career it might actually be jailor. Was there a prison here?
"England and Wales Census, 1841," database with images, FamilySearch (https://familysearch.org/ark:/61903/1:1:MQ1L-GHX : 24 May 2019), James Broughton in household of Silvanus Broughton, Gedney Hill, Lincolnshire, England, United Kingdom; from "1841 England, Scotland and Wales census," database and images, findmypast (http://www.findmypast.com : n.d.); citing PRO HO 107, The National Archives, Kew, Surrey
As Velma says, in the 1851 census James is living apart from his family, probably as part of his duties. His occupation is described as "Rural Police Officer formerly T/Jailor", age 35. His birthplace is given as Skillington, Rutland. This is slightly odd as Skillington doesn't seem to have been in Rutland historically. It's close but in Lincolnshire.
Here is James in the 1861 census age 44. Susanna has presumably died as he has a new wife. His occupation as police constable and birthplace of Skillington, Rutland identify him. His sons-in-law are what we now called stepsons.
"England and Wales Census, 1861," database with images, FamilySearch (https://familysearch.org/ark:/61903/1:1:M7B1-GMY : 24 October 2019), James Broughton, Doddington, Cambridgeshire, England, United Kingdom; from "1861 England, Scotland and Wales census," database and images, findmypast (http://www.findmypast.com : n.d.); citing PRO RG 9, The National Archives, Kew, Surrey
So I still haven't established his birth but he's not the JB from Kirkby, who appears to have died young in any case. He seems to have been born in Skillington which is near Grantham in around 1816/76 if the later censuses are to be believed.