Hacker News new | past | comments | ask | show | jobs | submit login

I don't believe that responsible web scraping is malicious, but my belief relies on the assumption that the web is meant to be open. That is, information put on the internet that is publicly available is considered free to access, store, and later retransmit. The original web was designed for researchers to share their work, while the modern internet built on top of that platform has other moral systems that don't necessarily agree.

Anyway, on a purely technical level, scraping of publicly available content isn't inherently bad unless you're asked to stop, or are scraping so quickly as to cause a service disruption by tying up the target systems. There is nothing malicious about generating normal traffic at the rate of a regular user. The animosity arises from what you plan to do with the data, and whether the entity you're scraping agrees with your usage.




Thanks, that's what seems obvious to me too that it's just public data and it's possible to collect it without overwhelming the server with requests. I just don't get why someone wouldn't want you to look at their website.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: