Yes, I think that's right: black is logically linked with darkness, and human beings, like other diurnal lifeforms, don't like the dark. The association is found throughout literature, from the biblical outer darkness, where there is wailing and gnashing of teeth, to the Black/Dark Riders in The Lord of the Rings, via traditional fairy tales in which a beautiful but evil sister is described as fair/white of face but black of heart.