Working with good people is the key to me. The problem with that is that it can change overnight when a senior leader leaves and someone new joins - I had one place that went from "great" to "awful" when a new CIO started and his culture started filtering down.
Yeah it's a difficult question. For me personally, the things you highlighted (good people, good cause) are the most important ones and I'm in the privileged position that I have a job where I like both those aspects. But it's kind of badly organized and that is a shame, maybe even because of growth.