Hacker News new | past | comments | ask | show | jobs | submit login

What stops a scraper from detecting Anubis and just removing "Mozilla" from the user-agent string?



That would allow you to specifically lock out that bot based on its user-agent string. That's the main problem with AI scrapers, many of them normally use user-agents that cannot be easily blocked, so other means have to be found to keep them off your grounds.


It breaks websites when you do that.




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: