Yet there are boundaries on which weapons we can and cannot develop: Nuclear, Chemical, Biological etc.

Indeed. Usually weapons are banned if the damage is high and indiscriminate while the military usefulness is low.

There is at this moment little evidence that autonomous weapons will cause more collateral damage than artillery shells and regular air strikes. The military usefulness on other other hand seems to be very high and increasing.

It seems like the sort of thing we shouldn't be wanting evidence of in order to avoid, though

Like skydiving without a parachute, I think we should accept it is a bad idea without needing a double blind study

Not all is bad, it's preferable to have autonomous systems killing each other than killing humans. If it gets very prevalent you could even get to a point where war is just simulated war games. Why have an AI piloted F-35 fight a AI piloted J-36? Just do it on the computer. It's at least 1 or 2 less pilots that die in that case.

It's a bit too late for that, since Ukraine and Russia are both already using AI-controlled drones in combat.

The risks needs to be weighed against the downside of not deploying a capable system against your enemies.

those are mostly drawn on how difficult it is to manage their effects. chemical weapons are hard to target, nukes are too (unless one dials the yield down enough that there's little point) and make land unusable for years, and biological weapons can't really be contained to military targets.

we have, of course, developed all three. they have gone a long way towards keeping us safe over the past century.