You don't need another complete set of servers: the new production servers, before beginning actual service, are available for brutal stress tests and all sorts of other experiments while the old servers running the old software are still in use and undisturbed. If the system is completely new and there is no old software on old servers, it's even easier.
Considering that software mishaps in securities trading can annihilate a company of any size, not only cost you as much as a bunch of servers, lack of testing appears very reckless.
> You don't need another complete set of servers: the new production servers, before beginning actual service, are available for brutal stress tests and all sorts of other experiments while the old servers running the old software are still in use and undisturbed. If the system is completely new and there is no old software on old servers, it's even easier.
We did not generally physically swap out servers each time we changed a line of code. Rather, we had two identical sets of servers, one for production and one for test. For test, we had some additional machines to run matching engines. If we wanted to make the test environment more closely match an actual venue, we would need many more machines that run matching engines. That still might not be sufficient to produce identical output timings in prod and test, though, because venues generally do not publish the sequence of all incoming ticks they receive during a day, the source code of their matching engine, what hardware they are using, their kernel version, and so on.
> Considering that software mishaps in securities trading can annihilate a company of any size, not only cost you as much as a bunch of servers, lack of testing appears very reckless.
It's strange that you replied to me suggesting we had no production-like test environment ~10 hours after I replied to you saying that this code did fine in our production-like test environment.
Considering that software mishaps in securities trading can annihilate a company of any size, not only cost you as much as a bunch of servers, lack of testing appears very reckless.