> If it's not "open and inspectable" then it cannot be reproduced by another group/researcher.
That doesn't follow. They can (attempt to) reproduce with the same hardware, firmware, and drivers. They can reproduce with similar hardware from another vendor (AMD). If you're using OpenCL, you can reproduce on CPUs, albeit much slower and probably only for partial results. Maybe you want to break out some pen and paper to verify a few calculations of the CPU for even more partial results because you don't trust Intel's firmware blobs, even though you've presumably cross checked against AMD hardware, but you've gotten quite silly at that point.
Maybe some of these changes won't give you bitwise identical results - but then again, neither does most science. You check if the broad results and patterns are statistically similar. Maybe they are, and the results survive a large variety of similar but different test setups. Maybe they don't, and you discover an uncontrolled variable, like the exact chemical composition of the surface of your glass beakers being contaminated by impurities, or your AI being sensitive to the exact floating point rounding behavior of your GPU.
If anything, you want to see if your results survive variations in the uncontrolled variables, to make sure you've properly determined what factors are controlling the results.
If you cannot reproduce then you do not know whether it is due to the equipment/materiel varying or not and you have severely reduced options for investigating that.
If you look at the problems with even reproducing and testing the bugs that the specific situation under discussion exposes it is illustrative of the difficulties involved with working with a non-modifiable, unexaminable platform.
In your example, the researchers would be forbidden by IP laws from examining and testing the surface of the beakers for contaminants and publishing a description which allowed other researchers to make similar modifications.
It's the most sub-optimal situation in which to obtain something which starts to approximate reality. Not impossible, just difficult, and likely to lead to shenanigans.
That doesn't follow. They can (attempt to) reproduce with the same hardware, firmware, and drivers. They can reproduce with similar hardware from another vendor (AMD). If you're using OpenCL, you can reproduce on CPUs, albeit much slower and probably only for partial results. Maybe you want to break out some pen and paper to verify a few calculations of the CPU for even more partial results because you don't trust Intel's firmware blobs, even though you've presumably cross checked against AMD hardware, but you've gotten quite silly at that point.
Maybe some of these changes won't give you bitwise identical results - but then again, neither does most science. You check if the broad results and patterns are statistically similar. Maybe they are, and the results survive a large variety of similar but different test setups. Maybe they don't, and you discover an uncontrolled variable, like the exact chemical composition of the surface of your glass beakers being contaminated by impurities, or your AI being sensitive to the exact floating point rounding behavior of your GPU.
If anything, you want to see if your results survive variations in the uncontrolled variables, to make sure you've properly determined what factors are controlling the results.