Hacker News new | past | comments | ask | show | jobs | submit login
Google boutiques.com has no robots.txt (boutiques.com)
6 points by techvibe on Nov 17, 2010 | hide | past | favorite | 5 comments



Well, you can try it with:

wget -O- http://www.boutiques.com/robots.txt -U "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

and you get the same error.

I tried a couple of other bot user agent strings from http://www.useragentstring.com/, same result


Why would it?


they have NOINDEX, NOFOLLOW robots META tags that tell search engine crawlers to go away.

Maybe it's their way of playing nice with all of the merchants they are listing.


Could it be user agent filtering?


So?...




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: