In theory that's a huge problem. In practice we have 1.X billion miles of data and it does not seem to be such a huge deal. Either the systems are already fairly good, or people mostly pay attention.
Granted, I am approaching this from the perspective of a more relaxing driving experience not necessarily from a pure safety standpoint. People spend 20,000+ hours driving in a lifetime making that less stressful is a huge benefit even if they are still stuck in their cars and can't get work done.
Granted, I am approaching this from the perspective of a more relaxing driving experience not necessarily from a pure safety standpoint. People spend 20,000+ hours driving in a lifetime making that less stressful is a huge benefit even if they are still stuck in their cars and can't get work done.