The question isn't who's legally liable. The question is are you doing things more unsafely than you need to be. You don't have to be legally at fault to be doing something that's unsafe for other drivers to deal with. Sudden breaking at highway speeds, especially for no actual reason, even if the person behind you is following at a safe distance, is rolling the dice that every car behind you for a while is driving at full attention.

"I require every other human driver to be driving correctly so that my AI-car may drive unsafely" seems like a bad bet. I'd also imagine that, if as a human, I just slammed on my breaks for no reason on the highway, I would be found to be at fault for an accident assuming the car behind me wasn't directly tailgating me.

You seem to view safety as some binary that doesn't account for frequency or severity of incidents. Your framing suggests you think it's better to continue with human drivers and the commensurate 40k lives lost each year than to use an AI that has even the slightest possibility of causing even the least significant accident irrespective of whether or not any accident occurs in practice.

I suggest it's better to compare a given AI with humans in terms of fatalities caused per million miles driven. If an AI performs a little better than humans it should be legalized and if it performs dramatically better than humans, it should be mandatory.

Of course, this is where we need more data and greater transparency so we can answer these questions.

I'm not trying to say that at all. What I'm saying is that we're at an awkward time now, where this sort of quick AI-assisted breaking is especially dangerous because fallible humans are most of the rest of the drivers on the road. In an all FSD world, this wouldn't really be a problem.

You're putting a lot of words in my mouth and assuming I'm against working on AI driving because one person might ever die. All I'm trying to point out is that it's pretty worrying to have a system that could cause a highway-speed accident because of a well-known and decently common bug. I'd be equally worried if it came up that some other decently selling model of car would randomly have the ABS system engage.

I wrote my OP here because the parent poster was casually talking about "not really break checking people" as if that's just normal behavior that's a part of R&D, instead of an AI accidentally emulating dangerous aggressive driving patterns that FSD is supposed to do away with. I'm not trying to ban FSD research or anything. I want this improved! It's just scary when people excuse dangerous behaviors by FSD systems because it's otherwise safer.

The other issue is that more data and greater transparency are both not things Tesla seems to have any interest in providing anyone, so while this may get fixed, it's not really pushing the industry forward all that much if no one other than Tesla is going to benefit. There's plenty of mentions in this thread of this sort of issue happening on other cars and adaptive cruise control systems that could benefit from an improvement for the betterment of all drivers, but instead "not breakchecking people" is going to be a unexplained feature improvement in some FSD patch probably.

if a car is tailgating me (less than one car length of space behind me above 30 mph), and my Tesla sees a ghost and brakes hard, and the car behind me rams into me, then that is completely on them. it's on me if they were further away than that.

Sure, but fault aside, you still just got into a highway-speed car accident. I don't get to decide if the person behind me is a reckless driver or not and "but the law says it's not my fault" doesn't do away with any injuries or damage to my car that happens because of it. There's plenty of completely legal things you can do that will create unsafe situations on the road, and you get a mark on your insurance for getting in the accident whether or not it's your legal fault that it happened (as I found out when I lost the front half of my car to flying road debris.)