Outside of the intended use-case of _replacing_ Datadog, I think this may actually serve as an excellent local development "Datadog Lite", which I have always wanted, and is something embarrassingly, sorely missing from local development environments.
In local development environments, I want to:
- Verify that tracing and metrics (if you use OpenTelemetry) actually work as intended (through an APM-like UI).
- Have some (rudimentary, even) data aggregation and visualization tools to test metrics with. You often discover missing/incorrect metrics by just exploring aggregations, visualizations, filters. Why do we accept that production (or rather, a remote deployment watched by Datadog etc.) is the correct place to do this? It's true that unknowns are... unknown, but what better time to discover them than before shipping anything at all?
- Build tabular views from structured logs (JSON). It is _mind blowing_ to me that most people seem to just not care about this. Good use of structured logging can help you figure out in seconds what would take someone else days.
I mean, that's it, the bar isn't too high lol. It looks like HyperDX may do... all of this... and very well, it seems?!
Before someone says "Grafana"-- no. Grafana is such a horrible, bloated, poorly documented solution for this (for THIS case. NOT IN GENERAL!). It needs to be simple to add to any local development stack. I want to add a service to my docker compose file, point this thing at some log files (bonus points for some docker.sock discoverability features, if possible), expose a port, open a UI in my browser, and immediately know what to do given my Datadog experience. I'm sure Grafana and friends are great when deployed, but they're terrible to throw into a project and have it just work and be intuitive.
Yes! We definitely do - in fact this is how we develop locally, our local stack is pretty intricate and can fail in different areas, so it's pretty nice for us to be able to debug errors directly in HyperDX when we're developing HyperDX!
Otel tracing works and should be pretty bulletproof - metrics is still early so you might see some weirdness (we'll need to update the remaining work we've identified in GH issues)
You can 100% build tabular views based on JSON logs, we auto-parse JSON logs and you can customize the search table layout to include custom properties in the results table.
Let us know if we fulfill this need - we at least do this ourselves so I feel pretty confident it should work in your use case! If there's anything missing - feel free to ping us on Discord or open an issue, we'd likely benefit from any improvement ideas ourselves while we're building HyperDX :)
Since my comment is too old to edit now - musing on this a bit more I think this would be pretty awesome to turn into a well-supported workflow to have a low-resource-usage/all-in-one version for just local development.
If anyone wants to chat more about this - I've kicked off an issue [1] to gather interest and everyone's feedback.
I use InfluxDB for this, it comes with a frontend UI and you can configure Telefraf as a statsd listener, so the same metric ingestion as datadog pretty much. There are docker containers for these, which I have added to my docker-compose for local dev.
I think it does log ingestion too, I haven't ever used that, I mostly use it just for the metrics and graphing.
In local development environments, I want to:
- Verify that tracing and metrics (if you use OpenTelemetry) actually work as intended (through an APM-like UI).
- Have some (rudimentary, even) data aggregation and visualization tools to test metrics with. You often discover missing/incorrect metrics by just exploring aggregations, visualizations, filters. Why do we accept that production (or rather, a remote deployment watched by Datadog etc.) is the correct place to do this? It's true that unknowns are... unknown, but what better time to discover them than before shipping anything at all?
- Build tabular views from structured logs (JSON). It is _mind blowing_ to me that most people seem to just not care about this. Good use of structured logging can help you figure out in seconds what would take someone else days.
I mean, that's it, the bar isn't too high lol. It looks like HyperDX may do... all of this... and very well, it seems?!
Before someone says "Grafana"-- no. Grafana is such a horrible, bloated, poorly documented solution for this (for THIS case. NOT IN GENERAL!). It needs to be simple to add to any local development stack. I want to add a service to my docker compose file, point this thing at some log files (bonus points for some docker.sock discoverability features, if possible), expose a port, open a UI in my browser, and immediately know what to do given my Datadog experience. I'm sure Grafana and friends are great when deployed, but they're terrible to throw into a project and have it just work and be intuitive.