> one's race doesn't impact other people negatively

Racists would say otherwise. If racists were a majority of society, would you agree that we as a society are right to govern that way?

You know, we also have tomes of legislation about what we don't regulate. In most Western societies, speech has pretty big tomes on that.