Technically you lose points every time you downvote someone, or you get downvoted, or if your best answer star gets removed the points that were rewarded with it are also removed. It doesn't matter much, and once you have the G2G6 badge it won't be removed :)