Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
Symbiote
on April 21, 2017
|
parent
|
context
|
favorite
| on:
Ask HN: Can I get in trouble for crawling using th...
It would be sufficient to let some requests come through from "Googlebot", and then deal with them (block, rate-limit, whatever) once the DNS check has been completed.
Consider applying for YC's Fall 2025 batch! Applications are open till Aug 4
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search: