as I understand it, cancer is essentially DNA damage. a piece of DNA gets randomly rewritten so that instead of doing its normal job it replicates and grows, using up resources and killing off neighbouring cells until it gets to the point where it prevents the organ/system/host from functioning
it's a little like a computer memory leak caused by a bit flip
cotton, silk, linen and wool are all harvested from organisms that contain DNA. if they promoted DNA damage, the organisms that produce them likely would have died off and been selected out long ago
of course humans could be vulnerable in ways plants, worms and sheep are not, or the damage could be slow enough to only affect something with a human-length lifespan, but there's a level of reassurance in knowing that those fibres come from something containing DNA
hydrocarbon-based fibres do not give me this reassurance
thank you for this comment, by the way. it forced me to question my thoughts and really helped me to a better understanding: a natural product is produced from something that has dna. obviously this isn't the original intention of the word natural, but it's an extremely useful explanation for an instinct, and your comment forced me to it