> Science gets done but the rewards seem to filter preferentially to those who are able to game the system, and the system exists out of a need to make one's self look as productive as possible
This is largely due to the current model of science funding, and not just in the US. Here in Germany, many involved in public (i.e. at an university, not inhouse r&d at a company) science only get 1-year-limited chain contracts with no real security and low pay, as many grants and funds are also available only in the same time frame which means it's a constant hustle for funding, especially at third parties.
IMO there is only one way to solve this problem: politics have to irrevocably allocate fixed chunks of money for public scientifical investment for long terms (think 10 or 20 years) to give the scientists and universities actual security in their planning and staffing. This would also solve the problem that e.g. NASA has with each President reversing course. No wonder that the last Moon visit was decades ago when the priorities get completely turned over every 4-8 years.
I agree that the root cause of a lot of it (although not all of it) is funding. I basically share your perspective about long-term funding. I personally would like to see indirect costs in the US eliminated, or much more heavily reduced, audited, and justified. I also think there needs to be dramatic shifts in funding along the lines of what you mention. Proposals have already been floated, by former federal funding heads no less, along these lines. Lotteries would be good, as would awards based on merit rather than application (Hungary's model of funding people based on bibliometric factors is a good idea, even if it runs into the problem of gaming citations as pointed out here). Longer-term funds also seem like a good idea, which is the whole idea of tenure in theory.
There are other factors at play too, that are harder for me to pin down. Funding models are a big problem, but there's something related to attention-seeking or metrification at play too. Some of this has probably always been around, but in talking to older colleagues I get the sense that things are much more splashy and fad-driven than they used to be, with much greater pressure to produce in volume. A colleague explained that when it's that much easier to write and publish a paper, there's more of an expectation that you do more of them, even though the idea development time isn't any shorter.
> This would also solve the problem that e.g. NASA has with each President reversing course. No wonder that the last Moon visit was decades ago when the priorities get completely turned over every 4-8 years.
This is also one of the arguments against promoters of so-called "term limits" for congressmen and others; imagine this kind of churn in priorities occurring with major and minor public works projects!
We don't have to wonder too much - we can already see it at the state level with governorships changing; one recent large change of this kind is with California's high-speed rail system. For all of it's "boondoggle-ry" and problems, I don't think the way it's been "axed" lately will be of help to completing it. In fact, it might just be a self-fulfilling prophecy for its opponents.
That's only one example; I'm sure others in other states could be easily found as well if one were to look. Ultimately, that kind of thing would only get worse with term limits on representatives to Congress, because federal funding for such large scale projects is needed - and that would end up likely in flux, and ultimately scuttle projects that depend on steady funding to be completed.
One could argue that individual state projects should only be funded by the state itself, but that notion of state self-sufficiency went out with the end of the Civil War. I also tend to wonder if - under a term-limited system - such a thing as the interstate highway system could have ever been built. It doesn't seem likely.
I'd prefer a 12-year term limit (for a 4-year period). That's 2 years to get up to speed, 8 years of working, and 2 years to pass knowledge on to the next generation. For long term projects the funds could be allocated at project start (e.g. via special-purpose bonds, just as in the early days of railway construction), so the project will be independent from political issues.
This is largely due to the current model of science funding, and not just in the US. Here in Germany, many involved in public (i.e. at an university, not inhouse r&d at a company) science only get 1-year-limited chain contracts with no real security and low pay, as many grants and funds are also available only in the same time frame which means it's a constant hustle for funding, especially at third parties.
IMO there is only one way to solve this problem: politics have to irrevocably allocate fixed chunks of money for public scientifical investment for long terms (think 10 or 20 years) to give the scientists and universities actual security in their planning and staffing. This would also solve the problem that e.g. NASA has with each President reversing course. No wonder that the last Moon visit was decades ago when the priorities get completely turned over every 4-8 years.