I think the change you mentioned (2000) has to do with maturity of the tech, as well as, maturity of the acceptance of this tech, as part of everyday life.

As far as the state of the web today, you can thank commercial entities. Yes, they did/do contribute to the web's existance, but at the same time, they make it much worse.