> it is "protecting" servers from web users who will not involuntarily provide commercially useful data/information
I don't think it comes down to that, I think it's more about the fact that your browser likely looks more like a bot than it does a human.
Also, rate limiting has a significant overhead and complexity at scale, where agent filtering is relatively cheap and easy to distribute. Though, this is largely a problem that has been resolved many, many times over and the additional overhead is not that bad. All said, I've met too many developers that don't conceptually understand public/private key encryption and would assume they'd mess up rate limiting.
I don't think it comes down to that, I think it's more about the fact that your browser likely looks more like a bot than it does a human.
Also, rate limiting has a significant overhead and complexity at scale, where agent filtering is relatively cheap and easy to distribute. Though, this is largely a problem that has been resolved many, many times over and the additional overhead is not that bad. All said, I've met too many developers that don't conceptually understand public/private key encryption and would assume they'd mess up rate limiting.