Right. And, sure, that's an edge case for most people. But even if such edge cases only crop up every now and then, you're now at the difference between a reliable automated door-to-door system and one that can drive you around most of the time but every now and then forces a hopefully sober/licensed/competent driver to get behind the wheel to take some actions.

That's a huge difference. Maybe you can address it with some sort of remote OnStar-like system but now you're forcing a remote operator to jump into an unknown context and take actions that were too hard for the AI.