Ellen was born at The Pinewoods Plantation in Wakulla County, Florida, on 22 Jan 1858. It is not definitely known who her parents were, but strongly suspect her mother was Mary Brown. She was enslaved by Elizabeth Grice Taylor and William Newton Taylor I. Taylor's wife kept a detailed journal of their moves from one plantation to another, including births and deaths of some slaves. Ellen is mentioned in a birth record, as well as the 1860 Slave Schedule (although the age difference is 2 years) in a list following Mary, George, and Fanny. In 1858, Mary is listed as having an infant, which may well be Ellen, as well as a son, George, and daughters, Fanny and Maria.