Your HUD isn't simulating a solid object floating in space.
If you want to hold a poster board in front of your face while driving, that's your prerogative, but that doesn't mean car or device manufacturers should design their interfaces with the assumption that you're being reckless. AR doesn't exist to fill the role of a smart watch and it never has (or Apple would have discontinued theirs when the Vision Pro came out).
I‘m multitasking either way, glancing down on the phone or watch for every notification.