The anonymity of votes is something that acts against the worst aspects of groupthink and knee-jerk opinions in the HN crowd. I don't trying to literally penalize disagreement with the median voter here would help at all.
The idea would be not to penalize local quantity (votes or downvotes on a topic) but aggregated quantities. Chances are that if I downvote 10 times per day (with say an avg number of down votes of 2 per day) that I am doing something wrong or I five times as much as the average user. In both cases it's food for thought.
Ratios of vote/downvote per user would probably also mean something.
Finally, I'd prefer to have these statistics private for each user, so there is no way groupthink could get a hold.
