Hacker News new | past | comments | ask | show | jobs | submit login

As I recall, this is outdated information. Internet Archive does respect robots.txt and will remove a site from its archive based on robots.txt. I have done this a few years after your linked blog post to get an inconsequential site removed from archive.org.



The most recent notice IA have blogged was in 2017, and there's no indication that the service has reversed course on robots.txt since.

<https://blog.archive.org/?s=robots.txt>




Consider applying for YC's Summer 2025 batch! Applications are open till May 13

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: