That's one thing Jon didn't touch on, the downvote. It's the easiest thing in the world for the giver, but to the receiver it feels like a kick in the gut. It definitely contributes to the feeling of hostility.
Personally I try to use my downvotes only as a last resort, for factually wrong answers or questions where clarification attempts went nowhere.
Has the idea of downvotes costing the user karma points been brought up before? At face value that seems like a good way to limit the amount of impulse or emotional downvoting. Most sites require a certain amount of karma to be able to downvote at all, but that addresses a slightly different problem I think. Having downvotes cost something (even if they're only fake internet points) seems like it would limit the idea of piling on. It would really change the dynamic so not sure what the result would be, so I wonder if it's been considered/discussed before?
[edit] I should have done my research before asking the question :P Apparently downvoting on S.O. does cost reputation/points. I guess the actual cost is so small it seems to be in the doesn't-matter category, at least for me as I didn't even notice. Perhaps if the cost was higher...
[edit2] I looked it up, and the cost is indeed trivial - one point. An upvote on your answer provides 10 points, and you can of course receive many upvotes for a good answer. The cost of downvoting is really only significant for a new/low reputation user. High reputation users can still be poor-downvoters. Seems to me there's a lot of opportunity to tweak the costs of downvoting to reduce the negative aspects of it.
I think downvoting is one of the things they got absolutely right over places like HN and reddit. When you downvote someone, you lose rep. And you can't downvote until you have a certain amount of rep, if I recall. (Or maybe once you hit 0 you can't downvote?) But it really makes you stop and think, and it makes serial downvoting much less of a problem.
I've never thought twice about the cost of downvoting a bad question or answer. I only downvote things I think are more harmful than helpful (incorrect or really-bad-idea or misunderstood-the-question type of answers).
It's the easiest thing in the world for the giver, but to the receiver it feels like a kick in the gut. It definitely contributes to the feeling of hostility.
I have several well recived answers there but I have never downvoted anyone because leaving it as is feels right than downvoting for the reasons you mentioned. I never asked a question there and tried figuring out the answers based on existing resources and on my own and it helped me learning a lot. This might not be the right way but being hostile and downvoting feels more than disagreement to me.
This is the reason why I avoid being there.
Personally I try to use my downvotes only as a last resort, for factually wrong answers or questions where clarification attempts went nowhere.