It absolutely does infringe on privacy. The only thing that is up for question is by how much and if the benefits outweigh the drawbacks. People who benefit from them existing are obviously going to have a different view point to everyone else (on average).

I don't think that referrers should ever have been a part of the protocol and I don't think that the commercial value of them existing should have any influence on whether or not HTTP continues to include them. Unfortunately, both Google and Microsoft benefit financially from the existence of referrer headers, so I don't see them going anywhere in Chrome or IE at least.