It is not only the algorithm which works so. Our brains also in a similar fashion; we first try to find out whether someone else's comment resonates with our impressions or fragments of what remains in our minds after reading the story.
So I upvoted you because you wrote first. You have to first mover's advantage. And I believe it is fair.
The first mover advantage goes beyond increased exposure. It means that one has maximumized their ability to shape the direction of the conversation in alignment with their thoughts. Because early comments are more likely to draw responses, the top poster in a thread has the most opportunity to clarify and extend their thoughts in follow-up responses.
Follow-up responses will tend to be received in the same vein as the initial post, and if the initial post gained traction the one's follow-up is likely to garner upvotes as well.
I tend to up-vote a good deal of comments, but -- like you -- I do it more the first time I read a post's comments. So, any comments written before I read it are more likely to get upvotes than any that are written later and which I see on later visits. (This is compounded by the fact that I'm trying to cut down my lurking on HN, since it's so easy to keep reading all the good stuff that other people write.)
So I upvoted you because you wrote first. You have to first mover's advantage. And I believe it is fair.