Yes, but if I really wanted, I could go into a specific line of code that governs some behaviour of the Linux kernel, reason about its effects, and specifically test for it. I can't trace the behaviour of LLM back to a subset of its weights, and even if that were possible, I can't tweak those weights (without training) to tweak the behaviour.
No, that's what I'm saying, you can't do that. There are properties of a Linux system's performance that are significant enough to be essentially load-bearing elements of the global economy, which are not governed by any specific algorithm or design aspect, let alone a line of code. You can only determine them empirically.
Yes there is a difference in that, once you have determined that property for a given build, you can usually see a clear path for how to change it. You can't do that with weights. But you cannot "reason about the effects" of the kernel code in any other way than experimenting on a realistic workload. It's a black box in many important ways.
We have intuitions about these things and they are based on concrete knowledge about the thing's inner workings, but they are still just intuitions. Ultimately they are still in the same qualitative space as the vibes-driven tweaks that I imagine OpenAI do to "reduce sycophancy"
