I'm sure there are a lot who wouldn't care, but MANY would leave (including myself), and recruitment would dry up. It would be a disaster.
Despite what hacker news comments would have you believe, Google is (in my strongly held opinion) a force for good in the tech world, and very genuine in what it tries to accomplish.
Becoming a patent troll would undermine that sentiment.