Hacker News new | past | comments | ask | show | jobs | submit login

Generally, any sort of async/batch inserts will get you decently far, but still will have limitations well before you get to million rows a second, mostly because it is really difficult to get your batch size large enough from individual producers without some sort of aggregation, which that aggregation is a challenge if you care about durability.

So often that means you need something like a Kafka to get the bulk ingest to really perform to get batch sizes large enough.

That kind of gets into one of the challenges of OSS observabilility systems, you don't want to make the dependencies insane for someone who only has a few thousand logs a second, but generally at some point of scale you do need more.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: