Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

it says it'll respect robots.txt if you don't want your page crawled (parsed? interpreted?)


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: