Almost everybody assigned the task of implementing this stuff (SSLv3, last century) didn't understand it, even for the old finite field DH where it's just about within the grasp of someone with high school mathematics if you insisted on having it explained before you implement. So it's just magic, and once one person does it wrong everybody else must do it wrong or lose interoperability.

And the "wrong" DH implementation works fine, except that it introduces a slightly larger side channel.

You mention a padding oracle, this isn't a padding oracle. It's an oracle because it provides the attacker with answers to questions they can't answer themselves, but there isn't any padding involved here.

And it's a pretty weak oracle, the insight you gain from asking the oracle questions is about a single pre-master secret, not the underlying DH private key or any long term authentication key.