I can't remember the exact detail, but one instance was a function checking whether a user should be paid based on some conditions. It checked the db, and I think because the codebase and db move fast, there was a new enum added a few months prior which was triggered by our transaction type.
So that helped function didn't account for the new enum, and we ended up sending >2 payments to users, in some cases I think over 10 to one user.
The issue was brought to customer support's attention, else we might have only noticed it at the end of the week, which I think would have led to severe consequences.
The consequences never reached us because our PM dealt with them. I suppose in all the financial loss instances, the business absorbed the losses.
> So that helped function didn't account for the new enum
This is where Scala/Rust's enforcement of having to handle all arms of a match clause help catch such issues - if you are matching against the enum, you won't even be able to compile if you don't handle all the arms.
The only db work I've done in rust required a recompile if the db schema changed, or even the specific queries your program used, because the rust types got generated from the schema. So in those cases the db change would have driven a rust type change and rust would have verified exhaustive handling.
Db changes are generally at runtime, how would you recompile rust code during the save of the data to the db? How do you rollback the change if a compile fails? How do you add the necessary code to handle new cases of the enum but not have it present in the db? This is amazingly interesting to me, would love to know more.
Maybe a code gen layer that generates rust types from a db schema. I don’t know rust but have seen those in other languages. I could see a DB enum type corresponding to a language specific enum type and then the language rules applying.
I do think this is a level of indirection myself; if the generated code was perfect and always in sync, that would be one thing, but by definition it is not the case.
So that helped function didn't account for the new enum, and we ended up sending >2 payments to users, in some cases I think over 10 to one user.
The issue was brought to customer support's attention, else we might have only noticed it at the end of the week, which I think would have led to severe consequences.
The consequences never reached us because our PM dealt with them. I suppose in all the financial loss instances, the business absorbed the losses.