No, the simulation harness and tests are open source and you can run them. It would be impossible for anyone to contribute anyway without it, for example, Snowflake, which heavily depends on it. It's built into the server binary directly, so the same code is always used, and it's simply a different operational mode when compared to the real server. I used to have a project to do lots of simulation runs on my big 32 core server and then aggregate the logs into clickhouse for analysis. It wasn't that hard.
However, they (at least at the time most of the developers were at Apple, many have now moved to Snowflake and the Apple team has grown a little I think) haven't released or integrated their nightly cluster and performance testing systems into open, nor have they integrated them with GitHub Actions or Nightly runs or anything. My understanding is that this is "just" a lot of compute cluster/platform orchestration code on top of the tests that exist in the repository. So, while Apple or Snowflake integrates changes across hundreds of concurrent fuzzing simulations on whatever platforms they have, if you write patches yourself, you're stuck with long simulation runs. Maybe that's changed; I haven't kept up since the 7.0 series.
In practice if you write patches and they accept them, they will just do the testing in their runs for you, on a cluster far larger than what you could have. Failures reports will tell you how to reproduce them from the test files. As a contributor, testing the system on your own is mostly a matter of how much money or how many CPU cores you can personally stand to set on fire.
Someone could probably integrate this functionality into a Kubernetes operator or something so that outside engineers could run large scale simulations reliably. But it is really expensive and CPU/compute intense, no matter how you go about it.
Those tests are not the implementations of the tests, just specifying the test case and the few options. But I found the implementations. I am not sure if this is all of the simulation tests, but it seems to cover the basic cases.
> Someone could probably integrate this functionality into a Kubernetes operator or something so that outside engineers could run large scale simulations reliably. But it is really expensive and CPU/compute intense, no matter how you go about it.
Yeah, that's basically an actually good implementation of the pile of crap that I threw together several years ago while writing a few patches. :)
And yes, I linked to the spec files because there actually isn't that much test code written in Flow I feel; the high-level specs in the .txt files can be mixed and matched so much to create a lot of variety from some small number of primitives, so that's really where all the good stuff is. Implementation vs interface, and all that.
However, they (at least at the time most of the developers were at Apple, many have now moved to Snowflake and the Apple team has grown a little I think) haven't released or integrated their nightly cluster and performance testing systems into open, nor have they integrated them with GitHub Actions or Nightly runs or anything. My understanding is that this is "just" a lot of compute cluster/platform orchestration code on top of the tests that exist in the repository. So, while Apple or Snowflake integrates changes across hundreds of concurrent fuzzing simulations on whatever platforms they have, if you write patches yourself, you're stuck with long simulation runs. Maybe that's changed; I haven't kept up since the 7.0 series.
In practice if you write patches and they accept them, they will just do the testing in their runs for you, on a cluster far larger than what you could have. Failures reports will tell you how to reproduce them from the test files. As a contributor, testing the system on your own is mostly a matter of how much money or how many CPU cores you can personally stand to set on fire.
Someone could probably integrate this functionality into a Kubernetes operator or something so that outside engineers could run large scale simulations reliably. But it is really expensive and CPU/compute intense, no matter how you go about it.
[1] https://forums.foundationdb.org/t/how-to-use-foundationdb-un...
[2] https://github.com/apple/foundationdb/tree/main/tests