Their associations are their works and words. You can't just handwave that stuff away as being irrelevant because you like the parts of their works and words that don't touch on anything you deem to be an association.
>I know people of the political party I tend to disagree with who are saints and dedicate the majority of their time to others.
Except for the part of the time that they are devoting to supporting a political party that is actively harming you and people you love.. you can't just wave that off as not being important just because they are nice the rest of the time.
>I know people of the political party I tend to agree with who are insufferable selfish pricks and treat others terribly.
And they should be negatively judged for the latter behavior.
Both examples should be negatively judged for their behavior, you're just choosing to ignore some of the behavior of the first person.