Unless we start chopping these tech companies down there's not much hope for the public internet. They now have an incentive to crawl anything they can and have vastly more resources than even most governments. Most resources I need to host in a way that's internet facing are behind keyauth and I'm not sure I see a way around doing that for at least a while
Certainly most of the ones with green names for whom a reply to me criticizing a big tech company or disavowing them in aggregate is among their first ten must be