Robots.txt User-Agent: \* Disallow: /x? Disallow: /vote? Disallow: /reply? Disal... | Hacker News

Hacker News new | past | comments | ask | show | jobs | submit

login

thinkbohemian on Sept 6, 2010 | parent | context | favorite | on: Remind HN: Go to Noobstories, flag spam

Robots.txt

User-Agent: * Disallow: /x? Disallow: /vote? Disallow: /reply? Disallow: /submitted? Disallow: /submitlink? Disallow: /threads?

sorbus on Sept 6, 2010 [–]

One assumes that a core characteristic of spammers and spambots is that they ignore Robots.txt - or at least I would be extremely surprised if disallowing them in robots.txt has any effect.

Consider applying for YC's Spring batch! Applications are open till Feb 11.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact