Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Is that really a thing? That must be such a hazard for their developers. I usually have a test for sites that I work on, that scrapes a few URLs as Googlebot, to verify that they are getting an optimized view (no JS, structural-only css).


God. That's the reason so many sites look great in the results and are confusing interactive messes when I get into them.


If it's any consolation, the site is well tested in Noscript mode.


You sir are a god among Web developers.

If only they all did this. So many sites I get to and they're a blank page or an absolute disaster....


Yes. Googlebot only crawls from legit addresses (even when their developers are trying new things) so it's an easy scraper/scammer signal to key off of.


No. Most websites don't do this.




Consider applying for YC's Fall 2025 batch! Applications are open till Aug 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: