Anonymity can't fix scientific peer review, it can only replace type I errors with type II errors. Instead of suppressing criticism that ought not to be suppressed, anonymity can (and often does) fail to suppress criticism that really ought to be suppressed because it is in fact false and defamatory. And indeed, what this article is really about is a lawsuit that alleges that this kind of error has in fact taken place.
Ironically, the very title of this article is a model of non-scientific thinking that ought to be subject to criticism, but attempts to inoculate itself against criticism by asserting that the subjects of the piece are protagonists "fighting to fix" a broken system. They're not. They're fighting to replace one broken system with a different broken system.
There is no question that scientific peer review is broken and needs to be fixed. But anonymity is not the answer. And holding anonymity up as something that should itself be beyond criticism is certainly not the answer.
I agree completely. In my experience as a PhD student (comp sci), all conferences make reviewers anonymous. As a result, there is very little accountability regarding the reviews you receive for your work. More than once, I have had papers rejected simply because a single reviewer barely read the paper and dismissed it. These kinds of reviews are very frustrating to receive, not only because they failed to understand the basic premises of your paper, but because these reviews contain no useful information on how to make your paper better for future submissions.
Of course, the opposite can happen where mediocre works slips through, and the reviewers that allowed that should be held accountable too. Its painful to me that so much of the acceptance process for research papers (in my field at least) is based on luck.
Moral of the story: reviewer anonymity is good but it comes at the expensive of accountability.
> Slips through? Mediocre work is the norm, or at least close to being the norm.
In computer science, this is likely because of the ridiculous number of conferences there are and how eager each one is to fill out their roster. I want to hear of the conference that published no papers this year because none were good enough for publication.
> I don't think it's based on luck, it's based on politics. Academic science is a racket.
I've heard (more than once) of people who submit a paper, have it rejected, and then get scooped by a professor who was on the PC of the conference that rejected them. There is a very real possibility of work being accepted or rejected purely because of what the reviewer thinks will be most beneficial to their career.
> In computer science, this is likely because of the ridiculous number of conferences there are and how eager each one is to fill out their roster. I want to hear of the conference that published no papers this year because none were good enough for publication.
Yep. But there is a reason it is this way, which is that to be successul on the academic feeding trough/career path, you have to serve on PC committees and you have to publish a lot of papers.
> There is a very real possibility of work being accepted or rejected purely because of what the reviewer thinks will be most beneficial to their career.
I think this is right, but it's less often so work can be scooped, and more often about building up your "camp." As a research professor, you need your area of expertise to be popular and you need to get a lot of citations for your work. You want to be a leader in an area that actually matters, where your definition of "actually matters" is actually purely herd mentality: what do other people consider to be important? So you want to reject work that bolsters your camp, cites you, etc., and reject work that poses a threat to the ascendancy of your area of expertise.
Agree with a lot of what is being said here. Have seen vicious anonymous reviews in CS - as we all have - both due to someone misunderstanding the work but also due to people from "a different camp" simply disliking a given approach. This must be common in other fields as well. Someone said on this thread that anonymity is not a guarantee of quality - completely agree. The move towards "open reviewing" for conferences in CS is very encouraging, however we will see how far it gets.
Minor reform of the referee system is treating the symptoms, not the disease.
The disease is that there is no honest, mutual, voluntary exchange between a party that values a given research project and the party that produces it.
Rather, we have federal bureaucrats handing out money taken from taxpayers willy-nilly via an old-boy network as described above. (The difference between taxpayers and slaves is that taxpayers only must sacrifice a fraction of their productivity to this absurd system, not all of it.)
In CS, all the research we do either goes to benefit shareholders of companies that ultimately profit from it, or (the majority) is just ignored because it's part of the paper mill competition. Those companies should be the ones paying for the research, not the taxpayers. Those companies are free riding, and "we the people" should put an end to it.
I'm not sure what the one has to do with the other. My research is corporate-funded and my papers still get reviews from people who didn't read the paper carefully. Unless you're proposing to eliminate the concept of publishing research at all...
What one has to do with the other is that the entire system is broken, here's my explanation for why the system is fundamentally messed up, and my explanation for how it should be. Simply trying to change refeering by itself is treating the symptom, not the disease.
> Unless you're proposing to eliminate the concept of publishing research at all...
I think putting my research on my website would be just as good as publishing it in a conference proceeding. I'm not against sharing research and having it be public, but I think conference proceeidngs and journals in CS are of very little value. If people want my research, they can find it on my website. And I think this generalizes... if I want your research, I can (presumably) find it on your website.
Ultimately, it may be useful to have some kind of aggregation of what research is coming out in various subfields. Internet fora and the like would serve the purpose fine. But sure, having a group of referees and some way to screen research and highlight what is good could certainly be useful. All that stuff should develop organically as it is needed, though.
The model we have now, with peer review, journals made out of dead trees, etc. is a hold-over from a pre-Internet time when a few people in every sphere of life controlled the information that was disseminated, because it wasn't possible for things to be open, since we lacked the technology. That is no longer an issue. Of course, academics will hold onto their little racket as long as they can (probably indefinitely).
I've had similar experience, and it can be tough, especially when the lack of interest from a reviewer is obvious. I still joke about the review I got that included the line "English need improving".
We ended up with 5 peer reviewers and an Editor override for one of our papers, instead of the usual 2 (I was the PhD student who actually did the algebra, so I saw all the comments.)
In fact, even after so much work to get it published, we now know of two errors in it - neither were pointed out by the reviewers of course.
I can't help thinking that that one experience sums up all of your problem cases above :)
Debatable on whether or not you can assume litigation is grounds for defamation. Even if it is true, it's not necessary that anonymity required the scientist to sue.
I think the appeal for the case to drop is not to discourage discussion of anonymity in review, but that it isn't discussion. A jury panel that decides to outcome of a paper could make science worse off than through a panel of peers.
The suit it a red herring. The point is that anonymity makes it more likely that bogus criticism will be heard, just as non-anonymity makes it more likely that valid criticism will not be heard. Anonymity is not a quality filter.
Ironically, the very title of this article is a model of non-scientific thinking that ought to be subject to criticism, but attempts to inoculate itself against criticism by asserting that the subjects of the piece are protagonists "fighting to fix" a broken system. They're not. They're fighting to replace one broken system with a different broken system.
There is no question that scientific peer review is broken and needs to be fixed. But anonymity is not the answer. And holding anonymity up as something that should itself be beyond criticism is certainly not the answer.