Typically, people who have a higher-than-average concern for the welfare of animals also have a higher-than-average concern for welfare of people. Compassion and empathy seems to grow for both, not be a trade-off.
Compare an analogous concern to the one you raise: "what if caring about people of color comes at the cost of empathy for your fellow whites?"
I walk past PETA’s west coast headquarters in Los Angeles every week, and they have inset windows behind a small ledge that’s raised from the street and protected from the weather [1]. They have the raised metal studs near the edge of the ledge, which are called “skateboard deterrents” [2], but given that the ledge is a couple inches high and unsuitable for any sort of grind trick, the purpose is clearly to prevent the homeless from sleeping on the ledge comfortably. My take on the vegan debate is that people who do it are making a good choice both practically and morally, and I wish I had the willpower to make it, and I also understand that PETA represents a very specific brand of veganism and animal welfare, but I always feel pretty sad when I see that PETA building and its unethical treatment of humans.
do you shed a tear when you walk past a homeless shelter that serves meat, clothes their homeless in wool and doesn't have bird safety film on all the windows? Oh you don't. When it comes to supporting animal welfare, you have to be considered "perfect" for some reason.
See my note below about spoon theory. I guess I could've been clearer that I meant not a total fix, it won't fix everything on it's own. I didn't mean that it's not part of the solution.
Compare an analogous concern to the one you raise: "what if caring about people of color comes at the cost of empathy for your fellow whites?"