> The fact is, these numbers are made up because the systems are too complicated to calculate based on individual components
No, you can perfectly well calculate component-by-component and come up with the 1 in 500 number. That's what NASA did (at great expense). What they failed to realize was this was only considering known-unknowns, and actual experience using the stack is required to learn about the unknown-unknowns which dominated the analysis.
This response deserves to be listed higher. I participated in risk assessment analyses, specifically for nuclear powerplants, specifically for extremely rare wind events. The nuclear industry is like the space one and has had to learn about unknown unknowns the hard way.
Yup. It comes up in other contexts too. In project risk assessment (“will we deliver on time?”) it is known as the inside view vs the outside view. The inside view is: here are all the parts we need to make, and how long each creation and integration step by itself should take, with known risks factored in. The outside view is: ok but the last 3 times we did something on this scale it took 2+ years when we thought it would take just 6 months.
You can use an inside-view analysis to get a minimum risk assessment. Something would have been very wrong in the analysis if we didn’t lose 1 in 500, since that’s the risk we expected from known factors.
And the outside view can give you a reasonable handle on maximums: objectively the risk of shuttle loss was about 1:75 or so, because that is what actually happened. If we were to have continued the shuttle program instead of cannibalizing it for the now defunct Constellation, we could have confidence that failures going forward would be at least 1:75 or better as we fix things, assuming regressions are rare.
The failure by NASA was in treating the inside view known-unknowns risk assessment as a maximum (“risk of loss is NO MORE THAN...”) when it was really a minimum (“risk of loss is AT LEAST...”).
No, you can perfectly well calculate component-by-component and come up with the 1 in 500 number. That's what NASA did (at great expense). What they failed to realize was this was only considering known-unknowns, and actual experience using the stack is required to learn about the unknown-unknowns which dominated the analysis.