In that case it seems like you could either find a way to try to break the attachment, or build more connections with (maybe new) people who do appreciate books as much as you do.
I get your pain, but unless you do something about it, it’s probably just going to continue or get worse.
Have you considered the fact that, if someone says, "My child has cancer and will die soon, it makes me so sad," and someone chips in with, "Maybe learn to be less attached to your kid maybe?" that this is clearly cruel. And it's a terrible kind of cruelty because its wrapped in ostensible compassion. It's the misuse of what is essentially a Buddhist principle to attack a person after they've revealed vulnerability. This is representative of a whole category of anti-social interaction that doesn't get enough attention or awareness. (This is similar to how it is underappreciated how the truth can be used to tell lies.) What makes this attack particularly pernicious is that if you push back on it, like I did, you yourself get attacked, because after all the person was just trying to help.
I get your pain, but unless you do something about it, it’s probably just going to continue or get worse.