>If I was generating image labels I absolutely would need to worry about that. However, since we're only generating images alone, we don't need to worry about bits of the labels getting into the output images.

Just to be correct, SD generates labels on images sometimes, so, we need to worry ;)