Maybe I'm saying something stupid, but couldn't the pirate bay be distributed as a torrent that is continually updated, and then you have a torrent client that can search the database?
Yeah, it's a clever idea but torrents are basically just content hashes, so they can't be updated by nature.
In fact TPB was already not serving torrent files in large part; every torrent with more than a handful of peers is stored as a magnet link, which literally is just a content hash. I'm positive there's something rather clever to be done there, but we can't put it into a torrent.
That's not updating the content of a torrent, that's updating a torrent file given a URL and a signature. It's not even BitTorrent, more like embedded RSS. Doesn't really help us out here.
The nice thing about BitTorrent is that if you point the new .torrent file at the same location as the old one, your client should incrementally download just the new parts. All you need is a way to distribute and switch over to new .torrent files automatically - something like BEP 0039, for example.
I've been thinking about this. Let's say we have a small html file, which loads the javascript torrent library. You keep this file on your desktop (and give it to friends), maybe even bookmark it to make it easier to open.
The javascript fires off, accesses a torrent (hard-coded into the html itself), which loads content for the website... images, js files, everything. It also loads the data (I'm assuming it's small, but I've read claims that the data is only about 90 megs) into javascript storage, ready to be searched locally.
The torrent swarm itself could become so big that it'd be self-sustaining, no original seeders would need to stay online vulnerable to legal threats.
To have the torrent continually update with new content, you'd have to break the protocol (which, for this one purpose, might not be bad). But I don't understand it well enough to know this is even possible for certain. With magnet links, you first have to download the whole torrent file before you can start downloading, and it may not be possible to coax it into get the updated torrent file again.
Even if you can't do that, updates themselves might just be new torrents... we can run javascript on this page, so we can be clever and have it derive new torrents/hashes to download. It would be good if anyone could update the data, for instance, adding torrents (though how to police that so they don't add junk or poison the thing... not sure).
The issue is that BitTorrent as a protocol converts hashes into files, but it's got no mechanism to associate a hash with some other identity. That means you don't need trust to operate it, but conversely there's no way make changes to a torrent after it's been seeded.
A system like Bitcoin+BitTorrent could do what you're describing, with Bitcoin filling in the missing distributed hash identification function. I don't really follow the distributed web scene, but I'm sure that's an area of active research.
the guys who developed bittorrent created a sync protocol. It's not open source though and people don't trust it. However this technique could lend itself to a bittorrent style protocol that continually updates.
It's a combination of people who have read-write access and people who have read only access but still contribute to the swarm. I think it's got potential.
