We understand the architecture but we don't understand the weights.

We also understand, down to a very very very microscopic level, how neurons work. We also know a helluva lot about the architecture of the brain. Does that mean we can explain our own intelligence, how our minds actually work? Nope.

No we don't. No it's not "magic". No we don't understand what the black box is doing.

For some values of “we”

For every value of we. "I understand the internals of GPT" is the fastest way to demonstrate you have no idea what you're talking about.