I read this as Declan saying that he and Baker were sufficiently acquainted for him to form a judgement of Baker's competence, with "he'd show up at parties I held" being to clarify the degree of their acquaintance. He's not claiming that they two things are causally connected.

And ignoring demagogues is not necessarily a good idea.