I’m relatively inexperienced in handling large-scale applications, but I’ve recently been tasked with scaling a web application for a large city. One major challenge I’m facing is logging HTTP requests efficiently.
Currently, we use Kibana and keep logs for 4 weeks, and we’re handling around 1 million logs per day. As more service are added and traffic increases, I’m concerned about storage costs, performance impact, and best practices for managing logs at scale. Most of the devs working here are also fresh out of school with limited experience.
Given that platforms like Twitter, Facebook, and Reddit handle millions to billions of requests per day, do they log every single GET and POST request, or do they use sampling, aggregation, or filtering?
What are the trade-offs between complete logging vs. selective logging, especially in terms of performance, storage, compliance, and debugging?
If anyone has experience working with large-scale logging systems, I’d love to hear how different companies approach this problem. Any guidance would be greatly appreciated!
You pay very cheap for storage (since it is stored in Google Cloud Storage behind), almost nothing, and relatively somewhat expensive for each data you extract of the system.
No need for the interactive exploration, you can do queries on a case-by-case basis, worst case you pay to scan one day of logs (or less if you pay Reserved capacity, you pay a fixed price).
0 devops needed.
If you want a self-hosted big database that is easy and scalable for that usage, check ClickHouse.
reply