Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

>Unless you do it from a Google Cloud instance, that is.

What's the reverse DNS for Google Cloud IPs? Google says to check that Googlebot's IP resolves to either a .google.com or .googlebot.com domain.

https://support.google.com/webmasters/answer/80553?hl=en



Couldn't you use GWT Mobilizer to scrape a site then index that?

Like this: http://i.imgur.com/ocR54Yq.jpg


Good point -- although it makes sense why it isn't frequently implemented. DNS lookups aren't cheap for this kind of thing.


It would be sufficient to let some requests come through from "Googlebot", and then deal with them (block, rate-limit, whatever) once the DNS check has been completed.


.googleusercontent.com




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: