Could Maria and Henry have 'divorced' and Maria stayed on in Blandford in their home on Bryanston Street as a single woman??? This would explain why Henry Milford was so 'late' in marrying his second wife Ellen Peters. See Maria Milford, born c1831 Dorset, with four additional children (apparently illegitimate). She was living on Bryanston Street in Blandford in 1861.
Marriage Henry Milford: Marriage. Event Date 24 Feb 1850. Event Place Blandford-Forum, Dorset, England. Marital Status Single. Father's Name; Mother's Name. Spouse's Name Maria Stanley. Spouse's Marital Status Single.   NOTE: Date also appears as both 24 February and 24 March 1850 in the parish registers for Dorset. Child: Catherine Milford.
Residence 1851 Census. Bryanston Street. Blandford, Dorset, England. Household. Henry Milford Head M 24 Blandford, Dorset. Maria Milford Wife F 24 Kingston, Dorset. 
Have you taken a DNA test? If so, login to add it. If not, see our friends at Ancestry DNA.