The issue isnt that the group is jerks (which is easy, you just stop and move on) but that they may bring down career ending social attacks on you. Unlikely with males even if superjerks.