people assign them to you when they see a post they like (green) or don't like(red), using the little scale icon at the top right of your post. based on your current status, it would appear as though you're comming across as annoying or an ass (the two most likely possibilities).
The blue icons are from people who have a negative standing themselves, and so their votes don't count.