I think it's not Google or google being gamed by SEO.. I think the web went from a cute jungle of documents into a boring market filled by baseless content.

Today my heuristic is: the older the HTML the better. Often the horrendously crafted website were the work either of naive individuals that had no hidden agenda beside having fun with webpages, or passionated ones that would put long articles full of interesting facts.

i would strongly disagree. I think there is far more good content on the web now than ever before .... but .... the signal/noise ratio is lower. The good content has increased but the crap has increased much more.

Google's original algorithm of giving weight to inbound links was a very good way of surfacing good content programmatically but I don't think they ever solved how to make it work in an adversarial environment.

Google's other algorithmic approach is to measure user response to sites (what do they click on, how long do they view it, does it satisfy their query, etc). that algorithm is great for maximizing revenue but is bad at surfacing new original content.

In the end they also had to hand curate, which is what failed to work before them, and what they originally replaced, so really we are back to DMOZ and Yahoo now. Just with a very sophisticated monopolist as the gate keeper.

Good point about the SNR.

The hand curation is coming back and not only at Google and there's a weird feeling that in the end we want "humans" as much as "results". Even the most satisfying algorithmic result don't have the same as a collection made by a group (considering that algorithm cannot pass some SEO version of the Turing test of course).