Hacker News new | past | comments | ask | show | jobs | submit login

That was my reaction. It seems like the article is saying two mutually exclusive things:

- We cannot block them because we can’t differentiate legitimate traffic from illegitimate traffic…

- …but we can conclusively identify this traffic as coming from AI crawlers.




It's a situation where it's difficult to tell for individual requests at request handling time, but easy to see when you look at the total request volume.


It's the behavior of the traffic in hindsight that's obvious. It's difficult to identify in the moment. This is by design.

Getting caught isn't a big deal. Getting caught in the act is. As long as they get their data, it doesn't matter if they're caught afterwards.




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: