Because most of the good sites won't let you crawl them any more, unless you're ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		Animats 10 months ago \| parent \| context \| favorite \| on: Two upstart search engines are teaming up to take ... Because most of the good sites won't let you crawl them any more, unless you're Google.

immibis 10 months ago [–]

Crawl them anyway. Adversarial interoperability is the name of the game.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact