It's the same reason free speech is something akin to sacred even for your worst enemies, because those who start taking away the bad people's speech are themselves always going to be one political actor away from having theirs taken away.