I don't know that there is a definitive answer for this. Slave ownership in the 1850s and 1860s is complex, as slaves were bought and sold regularly (which you see in the Virginia Chancery Court records for example)
I have an interesting case in my niece's family. I have been able to trace one of her lines back to slaves and a slave owner who were likely living in Danville, Virginia going back to the 1830s.
In this profile for Samuel Flippen, who was born into slavery around 1823, I have listed the slave schedules for 1850 and 1860 in conjunction with the 1830, 1840, 1850 and 1860 census entries for Thomas Flippen, the slave owner. While I can make a case that Samuel likely appears in the 1850 and 1860 schedules, I am cautious because I don't have any documentation listing his name prior to the 1870 census.
I also include references to a Dira Flippen, who is listed as being 100 years old in the 1870 census and is living two houses away from Samuel Flippen on the record. In the specific case of Dira, it appears as if she is in the slave schedules for 1860 and 1850, as there is a female slave who is much older than the rest of the slaves listed in these records. I hope to eventually create a separate profile for her, but I don't yet know how she fits with Samuel:
http://www.wikitree.com/wiki/Flippen-22
So - anyway, I don't know if this helps, but the general guideline I try to follow with slave schedules is to try and use the limited specific information in the schedules in conjunction with whatever other evidence I have been able to find.