> The other is that it implies serialised processing - you can't process anything > in parallel as you have single index threshold that defines what has been and > what has yet not been processed.
Fortunately Kafka is partitioned. You cannot work in parallel along partitions.
Also, you can streamline your process.
If you are running your data through operation (A, B, C).
(C on batch N) can run at the same time as (B on batch N+1), and (A on batch N+2)
Fortunately Kafka is partitioned. You cannot work in parallel along partitions.
Also, you can streamline your process. If you are running your data through operation (A, B, C). (C on batch N) can run at the same time as (B on batch N+1), and (A on batch N+2)
We do both at quickwit.