It's not always worthwhile to explain why you disagree with someone. If they say something so blatantly incompetent that responding feels like a waste of time, it's a lot better to just do the downvote and move on.
Other times, even if you do explain yourself, that person is just far too narrow-sighted to believe they may be mistaken on some point or other, or they're simply too arrogant to consider somebody else may be more knowledgeable on the subject. The worst offenders will just unjustifiably retaliate with personal insults and flame-baiting. Which as we know already will go unpunished, since the mods on this forum refuse to do their job.
It's a lose-lose situation. Downvote without explaining your reasoning, or respond to someone and have them verbally harass you. The best you can do is judge which is the lesser of 2 evils.