Reviewers could themselves have reputation levels that weight how visible their review is. This would make brigading more costly. There might still be a pseudoscientific brigade trying to take down (or boost) a particular paper, one that clusters so much that it builds its own competing reputatation, but that's okay. The casual reader can decide which high-vote reviews to follow on their own merits.