Is that true, though? [1] [1] https://webmasters.googleblog.com/2019/05/the-new-... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		adrianolek on July 2, 2019 \| parent \| context \| favorite \| on: Google’s robots.txt parser is now open source Is that true, though? [1] [1] https://webmasters.googleblog.com/2019/05/the-new-evergreen-...

groestl on July 2, 2019 [–]

It's a masterpiece that Google convinced everyone to use their crawler engine as a browser.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact