I used to think it was just poor comments, but it only happens on certain pages, which aren't at the top of my most popular list.

My theory is that the bots are working in waves -- first identify easy targets, then slowly exploit them. I wouldn't be surprised if the botnet guys are using Mechanical Turk-type operations just to work the CAPTCHAs and get valid logins for later use.

It used to be you could make generalizations about the bot wars. Now, with possibly millions of programmers out there with nothing to do but try to work the system? The complexity and nuance of attacks are several orders of magnitude greater than just five years ago. I'm not sure any kind of sweeping statement catches what's going on, except: there are a lot of entities on the web that are looking for doors -- any kind of doors. Once they find them, it may be months or years before they are ever used.

Now, imagine I am a spammer. I need to submit some automated comments that don't look like spam so that my spammy comments wll later get accepted. There are a variety of ways I can do this. The easiest is to give the user a generic compliment like "Wow, this was a great blog post! Thanks for publishing it." The comment doesn't add to the discussion but the blog owner is often reluctant to delete it because (a) it strokes his ego, (b) there is no way for him to tell if it is automatically generated or not. In fact, if his spam filter preemptively classified it as spam, he might even override the filter and reclassify it as ham; this would be a huge win for the spammer-to-be. Another way to do this is to automatically scrape a comment from another discussion about the page like Reddit, Digg, Twitter, or FriendFeed and submit it as a blog comment on the original page. A third way is to pay somebody a (very) small amount of money to read the blog post and write a comment (possibly using some fill-in-the-blanks response template system like those used in call centers.)

It works the other way too. If I want your comments to start getting flagged as spam then I should start submitting spam-like comments in your name. Then, users will start classifying my forged spam comments as spam and the automatic classifiers will start automatically classifying your valid comments as spam.

You might have a discussion in your blog's comments and then realize that everybody in the conversation is a robot except you. And you will learn something from those robots and/or lose an argument with them, when they get really good.

Eventually, you will be able to post a rough draft of your blog post, and the spam robots will start up a good conversation about it in the comments. Then, based on their feedback, you can revise it into the final draft (if the robots haven't already given you a link to a blog post that refuted your point or made it better than you can).

As a final step, we might notice that not only are the best comments are written by spam robots, but the best blogs are spam blogs too. Right now, people are blogging-for-adsense manually; in the future, Google will be able to blog-for-adsense itself, cutting out the authors.

Today, if a site wants its user-generated content (comments in this case) to retain its value, it needs to start filtering out the low-value content, regardless of whether it was generated by a spammer or a well-intentioned user. For example, If I add a comment to your blog "Great blog post!" on your blog, you should probably delete it, even if you know I am not a spammer. Unfortunately, if you do that, legitimate users who see their complimentary comments get deleted might react negatively (e.g. stop commenting or complain loudly, generating even worse comments). So, we can see that the spammers who are generating low-value comments already have the ability to drive away your valuable commentators if those commentators occasionally submit low-value comments.

we can expect comments that are generated by spammers to eventually become valuable enough that we want to retain them instead of filtering them away

What is the point? Why spammers would submit a valuable comment? Just to be kind of a door for them to enter through later, and nobody will think it's a spam?

Or, a spammer might generate some good comments just to get the comment classifier to mis-classify a blatantly spammy comment that will be submitted later.

Do you have any publications or a blog? I am very interested in learning more from you... you look like a scientist!

EDIT: But there is a point here that you did not really mention! You assumed that spammers will be smarter, but spam filters will still stupid? Don't you think that as spammers build a smarter bots, the spam-fighters will build a smarter spam filters?

Please re-read it again.

(This is an open offer to anybody: if you are the first person with your model of phone to email me, I will give you a free license if/when it works on your phone. Also notice how spammy this comment is.)

That's kind of a joke, but kind of serious too -- the entire idea of CAPTCHA is going to have to evolve in quite meaningful ways.

I liked your thesis, as speculative as it was. My general impression is that we're talking decades here, not years. Predicting out that far is always tricky. One can imagine receiving phone calls from friends -- the friends being electronically generated voice impressions of our real friends that try to sell us things. Once you start down this path of faking a person, you're going to end up in some very weird places. For instance, I could create "fake mes" that would interact on the web as well, creating blogs, commenting on articles, documenting a presence -- all for the purpose of leaving a bad trail for spambots to follow.

Imagine a time where we have a CAPTCHA that is impossible for people to solve, so that we only allow computers to submit comments.

Because it's spam!

In fact, the standard that any spam filter should rely on is: Is this spam or not? NOT is this good or not?

So even if the comment is really good and useful, then at least remove the link and keep the comment itself!

It could be a serious mistake if a spam-filter relies on "how much good this comment is", and this is another problem.

You make money out of being a spam, you don't make money out of stopping a spammer (generally, or at least nowhere near as much). Technology costs, hence spammers have a fiscal advantage in the spam wars.

Google bought postini and paid for it!

But I am sure that some hackers will be interested in fighting it and making a smarter spam filters, just the same as some hackers here discuss the issue here, and they won't get any money!

I don't have a blog myself (shocking, I know), but I have read that some blogs flag the mail adresses of previously accepted comments. The next time someone posts a comment, it will be accepted automatically.

Could it be that some spammers use this approach to save accounts for future spamming?

You submitted an article here which I read and found to be amazingly good. So I wanted to leave a short comment saying just that. I had to retry a few times, and got some pretty weird errors. I tried changing e-mail adress, subject, etc. to see if the error would go away. So your problem might be that there are a lot of errors on the submission process, so people try again, often using different names, etc.

Will look into that.

I like your idea about newbies trying out the system. Not as exciting as evil overlord botnets, but perhaps just as plausible. I guess the answers would be in a careful digging through the server logs.

Appreciate your trying. I just upgraded to MT 4.2, so hopefully the comment submission bugs have been worked out.

Appreciate your trying - I only did it because the article you wrote was really great :-) It's this one: http://www.whattofix.com/blog/archives/2009/02/technology_is...

I was seeing spam like this over a year ago across a network of sites all with the same set of messages (e.g. "Good site! Thanks!")

If your theory is right it would suggest that moderators set a higher standard (e.g., actual content to the comment) for what they permit?

Is there a solution to this in the works?

EDIT: Are you worried about the server? If you make a spam filter algorithm, then this is more likely to increase the load on the server, which increases the latency... and as a result, increases the headache?

But if we assume that they will take HN at any cost, then their course of action will match that of bacteria: attack, mutate, rinse, repeat. Eventually it will be either them or us that survives. And it probably won't be us.

Usually the price to spam into here is very very high because it is, by comparison, quite difficult.

Im tempted to agree with you - it is a spammer going after the good money testing a new bot.

Merely difficult? Isn't it impossible? Has a spam ever survived here?

Anyway, it depends on your definition of "survived" and whether there is value for the spammer in their story living on the new page for minutes or hours. Our company forums get an onslaught of spam about once per week, and though the spam never lasts more than a few hours (far less these days, as I've added a couple of moderators), it seems to be the same spammer doing it over and over again, so they must consider it worth a shot. When your employees time is almost free (or the work is done by a bot), pretty much any result is a worthwhile result.

I'm pretty sure the fact that spammers do something doesn't automatically mean it's worthwhile. There are some spammers whose stuff has been autokilled for months, but who keep submitting. They can't possibly be measuring the traffic they're getting.

The last request for spamming HN I came across was worth about $1,500 and up for a front page spot.That's a lot for a single link.

I will email it though (you will need to register on a forum and make a few posts I think :) been a while since I joined) if you wish. Which email goes direct to you?

Though I suspect you were attempting to call a bluff ;) oops...

    only comment without links up to 10 karma, 
    submit at 20 karma, 
    comment with links at 50 karma, etc. 

Initially one proves themselves in discussion, then they can bring new ideas to the table. By setting the limits at relatively low levels it doesn't discourage new participants.

As a relatively new member of the community, I believe that this is important because you don't want the community to become static any more than you want to get overrun a la digg, reddit, etc.

A few days registration won't work, though, because the bad guys will just start setting up accounts, keeping them dormant for the waiting period, and then do their dastardly deeds. Besides, karma is a better measure than seniority.

This is similar to what I wrote here: http://news.ycombinator.com/item?id=506028 Users should have a history record.

But it won't be an effecient solution, according to briansmith's approach.

Spammers could hire users to comment. Here is how I imagine it: 1- Spammers crawl the submitted article.

2- Ask real users to read it and comment on it.

3- Copy what users said and submit it automatically here.

4- You will think it's NOT a spam, and you will up-vote what the spammer commented, and this will allow the spammer to submit content and the problem won't be really solved/ but reduced.

Here is how it works, just one of these 2 options:

1- Spammers automatically create a blog, crawl the submitted articles here to their blog, and wait for a real user to comment on it, and then submit that comment here.

Or: 2- Spammers would create a bot that crawls the submitted article, submit that article to any public social news website, and wait for real users' comments, and then automatically submit the real users' comments here, and you will think it is NOT a bot/spam, but it is.

EDIT: This is something that can be implemented these days, not after a decade!

If this happens, will you think that this is a bot or not?

I agree that there will always be gaming strategies. I just think that having gradually increasing rights/priveleges at quick incremental karma steps is a good way to combat spammers.

When a new user comes to this site he will try out the functionality, click around and see how things work. One of the things that he will probably want to know is how to submit an article. So he goes ahead and tries it, using whatever link he has handy.

I know this because I did it myself when I joined. I deleted it right away though. I've seen the same thing on other sites, so it's not that unusual.

Edit: tuned down the wording a bit, since PG's reply indicates that this might not be the sole reason.

Can you see whether the users that do this end up being good citizens? That would shed some light on whether these are malicious accounts or just new users trying things out.

  - Step one is getting your bot to reliably create accounts.
  - Step two is getting it to create accounts and post links.
  - Step three is feeding it a list of 5000 of your sites.

Even I provided a significant solution to solve the problem: http://news.ycombinator.com/item?id=506028

Guess what? I got down-voted on my comment!

EDIT: If pg has no time for it, then why does not he allow us to code a solution, and he can review it? And if he likes it, then he would use it!

Your words made me feel really great.