I have a feeling we will see a resurgence of some of the ideas around expert systems; current language models inherently cannot provide guarantees of correctness (unless e.g., entire facts are tokenized together, but this limits functionality significantly).
My sense from the few projects I've seen attempt to use Druid is that there is quite a lot of infrastructure overhead / DevOps support required to manage a cluster at scale, and that fairly complex ingestion pipelines are required to load the data in the right format.
Anecdotally, I've heard that ClickHouse is easier to deploy from this perspective with similar performance, but would love to get others views / experience with these and similar data stores.
How is the documentation for this deployment and tuning? Last time I checked I had the impression that anything about the dozen of different node types weren't very clear, not to mention details about the ingestion were all over the place.
And it was hard to find examples of configuration, ingestion other than basic tutorials
Strange - we use Druid at scale at work, and it doesn't really require much maintenance. In fact, for the most part aside from some incidents caused by our own misconfiguration, it buzzes away just fine. Runs on Kubernetes. Most annoying thing was tuning segment compaction.
It's actually quite nice - because since everything is stored on GCS/S3, it is mostly self healing, we can treat the historical as cattle and not pets.
We also run clickhouse, and unfortunately the above is not true - at least in our setup.
This is one of the reasons why we chose ClickHouse as our backend at PostHog. We wanted something that was relatively simple to operate for our users who deploy PostHog on-prem. We've been super happy with it so far. Still not turn key in many ways, but it's been pretty great.
I run a ClickHouse cluster. I’ve heard Druid is more difficult, but ClickHouse isn’t exactly a piece of cake. It’s great for a while, but sometimes you can get bitten by weird states. Still, compared to everything I’ve used, for the use case where it excels, it really excels.
Clickhouse benefits from the ability to get started as a single binary (and even includes a clickhouse-local binary to do ad-hoc analysis on CSVs on your laptop). There’s only one node type, as well. It’s simpler and easier in that sense.
Running it at scale is different. It includes everything you need, and it’s not horrible of course - but there are certainly a lot of sharp edges to be mindful of.
Druid is horizontally scalable by itself if you have access to something like S3 or any compatible Object Storage. Druid's core design is remarkable, designed from the ground up to optimally leverage and work harmoniously with cloud tech. Once it is setup appropriately for your use-case, it's trivial to stamp out over and over with Terraform.
Several years back, I was running it on a single server along with Kafka ( I was requested to keep everything related to analytics to a single huge server ) and it started with a fight for zookeeper between these two. While it worked quite well thereafter, keeping it up was a battle.
Maybe situation is better with clusters and Kafka moving away from zookeeper.
I feel like it shares a lot of the complexity with the rest of Hadoop-adjacent products. If your company already manages standard Hadoop infra, it's probably not too different, otherwise it seems a quite bumpy road.
I think that the main issue is less the technique (although… yes, please use RL if you can) and more the lack of data. Browsing gives very little insight: dwell-time is a poor proxy for interest, and mixes horrid ideas that are so bad they are worth sharing with friends and confusing photos where you need to squint to figure out if it’s what you are looking for.
Both e-commerce and social media are really not good at gathering express feedback for what people want and valuing that expressly. Please, let me tell you that I did spend time looking at this thread about the latest reality TV scandal but I don‘t want to hear about it ever again! Please, let me tag options as “maybe” or let me tell you what you’d need to change for me to buy that shirt. Public, performative Likes and Favourite lists that are instantly reactivation spam-fodder… Come on, you know better.
I used to work for a big e-commerce site (the leading site for 18-25 y.o. females). We had millions of references (really) and it was a problem. The search team had layers upon layers of ranking algos, incredible papers at conference… but still, low impact on conversion. It was more than anything else that we could do, but nowhere as transformative as it could be. Instead, I suggested copying the Tinder interaction in a companion app:
* left, never see that item again;
* right, add it to a long list of stuff you might want to revisit. We probably would have to separate that from the Favourite list to avoid clutter, but maybe not, to make that selection worthwhile.
The learning you could get from that dataset, even with a basic RL algo to queue suggestions… People thought it was “too much” which I’m still bitter about.
How are bandits used in consumer choice problems? Bandits solve almost the inverse problem: which choice to offer/take when it's uncertain which is best, but the problem under consideration in the blog post is about predicting which choice a consumer will pick, a standard marketing problem.
