How hard have people pushed this thing? We get regular threads of effusive praise, but little criticism. Last time I mentioned that years ago my colleagues found half a dozen ways to lose data in FDB I got called out here and even in private emails, but it seems more valuable to know where the limits of these systems are, and not very valuable to read the positive feelings of people who used FDB in trivial and uncritical ways.
Yes, there are definitely a lot of big companies that have used FoundationDB very hard at huge scale for many years. That said, yeah, it feels like there are also a lot of folks on HN who just jump on the "cool, fault simulation" bandwagon and don't have a lot of personal real-world experience.
What I can tell you, for sure, is that if you find an issue with something as important and fundamental as data loss the team working on FoundationDB would take it super seriously.
FoundationDB is used at Datadog as the metadata store for Husky, the storage and query engine powering a significant number of Datadog products, such as logs, network performance monitoring, and trace analytics.
I know there are multiple companies that use it. The question is not whether people put things into FDB. The question is whether anyone has checked to see if their junk was still there later. I don't consider large scale deployments to be proof of anything. When I worked on Gmail we were still finding data-loss bugs in either BigTable or Colossus regularly, even after those systems had been the largest datastores on the planet for many years.
The time at which my colleagues found easy ways to lose data was well after Apple had claimed to use it in iCloud at scale. So, I don't think deployment at scale is a proof of correctness. The thing that needs doing is regularly looking in the database for things that should be there.
I don't recall any of those details but the test involved injecting a bogus block device that always returns garbage, and noting that this results in garbage records returned from client queries. And I don't think those kinds of issues have been eradicated, browsing through their github issues there are people trying to recover corrupted clusters. https://github.com/apple/foundationdb/issues/2480
