Same here! I don’t get it. It’s much more lightweight, fully FOSS and their custodian company (Synadia) is very friendly and open to self-hosting. Basically, they run a global super-cluster of Nats which is pretty much vanilla afaik.
My main gripe is lack of “best practices”. You’re pretty much on your own when it comes to important decisions like subject namespacing and some operational stuff can be a bit tricky, like changing options and migrating. They had “nats by example” but that seems to be quite out of date and not super helpful. They have started doing more videos which are great but not that much content overall.
Their auth story is very sophisticated but perhaps a bit over-engineered. Some use cases like having anonymous “self-signed” client keys are tricky to work with.
Other than that, it’s an amazing piece of tech. One that is scalable in a way that just makes sense and gets out of the way when multiple machines are involved.
Nats is simple because it doesn't bother addressing some of the scalability problems that Kafka was designed to address, like clustering with partitions:
You linked to NATS Streaming, which has been deprecated for years. It’s not relevant. It can still be used for legacy code that still needs it, as the URL even shows, but it is not recommended for new applications.
The current standard is NATS JetStream, which absolutely addresses scalability, arguably to an even larger degree than Kafka, since it can be used to create a globally interconnected supercluster of clusters where messages can be configured to flow between regions as needed. I doubt anyone would run a Kafka cluster that spans multiple regions.
I doubt anyone would run a Kafka cluster that spans multiple regions.
Stretch clusters and clusters with brokers in different AZs are absolutely a thing in Kafka. I make no comment on the ease relative to Nats, which I don't know at all.
I was talking about different regions altogether, not just different AZs. My understanding is that it’s generally a bad idea in Kafka due to the latency, but I could be wrong.
EDIT:
Definitely sounds like there are some tight latency requirements[0]:
> A stretched 3-data center cluster architecture involves three data centers that are connected by a low latency (sub-100ms) and stable (very tight p99s) network, usually a “dark fiber” network that is owned or leased privately by the company.
100ms would theoretically be enough to span the contiguous United States, but the references to "very tight p99s" and "dark fiber" make me wonder if 100ms is actually acceptable, or just a theoretical maximum that can only be allowed under absolutely perfect conditions. Either way, suggesting the user should have access to dark fiber between their regions does not fill me with confidence about the robustness of this solution. I'm sure it would be fine for geographically close datacenters, as AZs are designed to be.
