The statements made in the original post are so foreign to me. It sounds like the author is digging really deep to try and say something nice about the peer review process. To be fair, it might be true for CS where they are stuck in the weird trap of publishing in conferences, but in other circles everything goes into journals, and in this case the peer-review processes is definitely not there to "help the authors".
The peer review is there to help the journal maintain its reputation by preventing the publication of sub-standard stuff. Period. Sub-standard can mean uninteresting, incomplete, poorly written, or whatever the journal is aiming for. It is not there to safe guard the integrity of the literature against erroneous results...it's purely self-interest on the part of the journal.
In reality, a rejected paper will just be submitted elsewhere until it is eventually accepted. The authors cannot afford to spend 1-2 years worth of work on a project then have nothing to show for it, just because a reviewer didn't "get it". So authors will keep submitting it (hopefully with some improvements based on past reviewer comments, but maybe not) until it "gets through" somewhere, and eventually nearly everything gets the seal of "peer review".
> There is SO much more I could write on this subject, but I'm trying to stay on point (-:
Right, it's a low bar, and it's meant to be a low bar, and that's fine.
It's not no bar. Peer review adds some credibility to a paper. And the venue does as well. It's just less credibility than the popular imagination assumes.
This had a refreshing degree of realism. Naturally, any perfection-minded person will be offended by that.
A decent journal will stop using reviewers who do a lousy job, and complaints by authors about their paper's reviewers ought to be listened to and not dismissed as sour grapes.
> A decent journal will stop using reviewers who do a lousy job, and complaints by authors about their paper's reviewers ought to be listened to and not dismissed as sour grapes.
As a data point in ML conferences, I had a reviewer give me a strong reject with only a few sentences. First was a generic "not novel" with no further explanation. The rest was a complaint that my paper had serious flaws and pointed to an intentionally redacted link and a broken cross-reference. We reported them to the AC and the other two reviewers were borderline and weak accept, but both had low confidence (of course the inane reviewer was high confidence). The result was that the reviewer's rebuttal was much more angry and the weak accept said "authors addressed my concerns, but I've decided to lower my score."
I tell this story because at a certain point it is not a problem with bad reviewers, it is a problem with bad management. The most important thing that a system can have is self correcting behavior. A system that gives a bad faith actor a slap on the wrist and allows them to then escalate their bad behavior and influence others is not a well functioning system. Granted, this is just one example, but I've seen many others (such as a theory paper being rejected for lack of experiments). I've started collecting author tweets I find about these kinds of situations, but I think there's a lot more that aren't easily conveyed on social media.
I do really want to promote discussion of these things because I think we should always be improving our systems. You're right that nothing is every perfect, but we can still recognize flaws without being consumed with the impossible goal of perfectionism.
was shitcanned by a Microsoft reviewer (if you're the one and you're reading this: GFY)
First of all, he ignored it for six months, and it wasn't until I begged the editor that he got it moving. Then he gave one round of critique, and when I addressed it, he said, "the patent law changed" and closed the process. He knew nothing about patents.
As I said, the editors should not defer to these douchebags. Fortunately I don't need pubs for my career, but some people do.
Funny enough, none of my first author works have been "published" but since they are on arxiv I have a competitive citation count and h-index (most cites are from my first author works). Only group upset at me is my grad school. It's extremely frustrating because I have all indications of doing good work and clearly the community agrees. I think the funniest part of it all is that I've offered to show them my reviews and so we can discuss if I am doing something wrong or "just having bad luck." But no one wants to even entertain the idea that the system is dysfunctional.
I will never understand why in CS we use a conference system as our indicator of merit. It is an antagonistic zero sum game. There is no recourse for giving a bad faith review and you are incentivized to reject works. It's easier to reject and thus less time consuming -- every paper has flaws and limitations, just point at them and ignore context--, you marginally increase the chance of your own paper getting in, and venues use acceptance rates as the main measure to indicate their level of prestige (if I cared enough I'd just get LLMs to spam papers to abuse this). Every part of the system that does something good and beneficial to the scientific community relies entirely on people acting in good faith and against other incentives they have, favoring the purity of science. That clearly doesn't scale and clearly sets a stage for bad actors to overwhelm. I think we only continue this because they've successfully convinced good faith actors that there is no other way and that it isn't as bad as it is. I just wonder how much money and time is lost due to all of this (before we even account for the ridiculousness of charging for what arxiv does for free. The paper is given to them for free as well as the reviewing is done for free, and much of the organizing committee and all that is also free labor. Something this important shouldn't have so many similarities to a scam that seeks to extract money from the government and universities).
I am also in a conference-centric CS subfield, but I have published quite a lot in journals as well (both because of multidisciplinary collaborations and because my country has coarse-grained metric-based evaluation systems were conferences don't count much, so at least a part of my work has to go to journals to appease the system).
In my experience, journal reviewing is much worse, and especially much more corrupt, than conference reviewing.
At conferences you get bad actors with almost no oversight or accountability, true. But at least they typically don't have an axe to grind, because accept/reject decisions tend to be binary (there is no "revise and resubmit", at most optional recommendations for the final version that aren't effectively enforced).
At journals, the "revise and resubmit" option gives reviewers and editors a lot of leverage to push papers their way, and I have very often seen reviewers tacitly and subtly hint that the paper would easily be accepted if the authors included three or four references to papers by Author X (which are irrelevant to the subject matter, but of course, Author X will be the reviewer or a friend). Sometimes it's clear that the reviewers didn't read the paper, their sole interest on it is that you include their citations, that's the reason why they accepted to review and all they look at. Editors could in theory prevent this, but often they just echo reviewer comments, and sometimes they are complicit or even the originators of the corruption (it's also typical that the editor asks to cite a bunch of papers from the same journal, even if they're not related, because that inflates impact factor). In any of these cases, since the author is in a position of weakness and there's always the career of some PhD student or other down the line, one doesn't typically dare to complain unless the case is really blatant (which is typically not, because they know how to drive the point subtly). It's easier to go along with it, add the citations (knowing that they will lead to acceptance) and move on.
This has happened to me very often and I'm not talking about shady special issues in journals that print papers like sausages, I'm talking about the typical well-regarded journals from major traditional publishers. In conferences I've gotten all sorts of unfair rejections, of course, but at least the papers I've published accurately reflect my views, I can stand behind them. In journals, maybe half of my papers have this material that doesn't make sense and was added to appease a corrupt reviewer.
I find that many CS authors who haven't had to publish in journals have a "grass is always greener" mentality, and expect that if we moved to journals we would find a fairer review process... and if at some point we do so, they will receive a blow of reality (not saying it's your case, of course. There can also be people who have published in journals and disagree with me due to different experiences! And there are some journals that don't tend to engage in that kind of corruption, only that there are not many of them).
Yeah I don't submit to journals often but have had similar experiences. They have also asked for addition of significant experiments with significant compute. Like they wanted us to try our technique on an additional 3-5 networks (which are all of the same general architecture we originally did) and it is very clear that one of the reviewers was an author of one of those (but adding the other two to "mask" themselves). It would have doubled the project's compute and they weren't happy when we responded that we simply don't have the budget and aren't convinced the experiments would be meaningful but we'd be happy to add them to our table for comparison. All three suggested that the paper did not have much value but the paper already had >200 citations via arxiv by this time too... (several works built on ours and even got published...)
It's a really weird political game and I just think we need to move to a system where we either have reviewing seen as a collaborative/allied effort (as opposed to adversarial) or we rethink the whole system entirely and let reviews happen naturally (i.e. submit to OR and allow comments). It's just very clear that a system can't scale if it relies on all members acting in good faith. Especially when there's high amounts of competition.
Edit: one thing I find interesting is I contextualize different than other reviewers. Like I see a experiment section where they have a node of 2080Tis and think "okay, students or small lab. Compute bound, so do the experiments they performed optimize under those conditions?" Whereas I think most reviewers don't contextualize in this manner and so I see many act as if every lab has access to many A100 nodes. I think this matters because the small labs can still do essential work, but we just need to recognize the signal may not be as strong. Their workload is probably higher due to lack of compute too (and I subsequently tend to see deeper analysis and less reliance on letting the numbers do all the talking). I don't think the GPU poor "aren't contributing" because they can't, I think they can't because gatekeeping. It's insane to expect academics to compete with big tech in a one-to-one comparison. If all you care about is benchmarks then compute always wins.
No I just meant how much could it have changed since 2014 if it was also 2014 or soon after it, seems like a weird thing for a reviewer to say without further qualification
I'd have to go ask my dad about how all this works specifically lol
This was what a few academics hypothesized and set out to prove in what is now known as the Grievance Studies Affair [1]. Not only did they get multiple hoax papers published, they got recognized for “excellent scholarship” [2].
That grievance studies thing was idiotic, and really ought to be dismissed and forgotten. Reasons:
1. The hoax authors didn't submit gibberish or anything, like with Sokal - they submitted coherent papers that they considered absurd, but which they expected the journals to find reasonable. As such, they only thing they can even pretend to have proven is that they and the journals disagree about what's reasonable.
2. IIRC, every one of the initial hoax papers was flat-out rejected! If the authors had any backbone they could have stopped right there, and published a paper called "These grievance journals aren't as bad as we expected." :D But they didn't - they set out to iterate on each paper until a journal accepted it. And if your experimental methodology is "keep trying different things until X happens", you can hardly pretend to have proved something when X happens.
I mean, I agree with the premise the hoax authors were trying to prove. But their "experiment" was silly, and I think it's dishonest of them to pretend it proved something. Meh. Meh!
Even so, it was a good demonstration of the point above: the peer review process in journals is not designed to keep out garbage, and is incentivized to apply less scrutiny when the subject is politically convenient.
Peter Thiel in this recent talk argues by inference that this problem is likely deeper and more pervasive in hard sciences than in the humanities. https://youtu.be/lO0KH-TgvbM?t=22m20s
That's my issue with it, really - I think it was widely presented and received as having "demonstrated" something, but if you look closely, by reasonable standards it didn't.
Actually if you dig up the authors' initial article about it, IIRC at the end there's a line like "so what does any of this actually prove? Well, we'll leave that for you to decide." If the authors can't say what their experiment proved, I think it's reasonable to conclude it didn't prove anything :D
Any evaluation metric is subject to political influences. Whatever the exact mechanism, it's people who are evaluating the papers, and people have political views.
Excellent summary. I’d also mention the lack of a control group. We’re supposed to conclude that it’s easier to (eventually) get fraudulent results published in so-called ‘grievance’ fields than it is in (say) physics. But the hoax authors never made any effort to publish fraudulent results in those fields, so we can’t know if the percentage of papers that were eventually accepted would have been lower or not.
We were typing at the same time - see my other comment (sibling to yours) linking to Thiel arguing things are likely worse in hard sciences, and encouraging people to find ways to expose it.
>It was later reported that the manuscript included plagiarized sections from Chapter 12 of Adolf Hitler's Mein Kampf (My Struggle), in which Hitler describes why the Nazi Party is needed and what it requires of its members. The authors replaced Hitler's references to "National Socialism" with "feminism" and "Jews" with "privilege".
Isn't this a pretty good example of how crude the hoax was? Of course some things that Hitler said won't be awful if you literally change the key words in the sentences. Imagine if I played this trick the other way round. I take something innocuous that a politician has said and replace random words or phrases with 'National Socialism' and 'Jews'. Suddenly what they're saying seems really controversial! And this shows...what?
> Imagine if I played this trick the other way round. I take something innocuous that a politician has said and replace random words or phrases with 'National Socialism' and 'Jews'. Suddenly what they're saying seems really controversial
I am not sure this is obvious or even true. Can you provide one example? "Gas all the <X>"...what? capitalists? murderers? rapists? pedophiles? It doesn't even make sense for them, what else is there?
inb4, no you don't want to gas these people, you want to fix them if possible or jail them otherwise.
“We must fight against the influence of special interests in American politics.”
“We must fight against the influence of Jews in American politics.”
You may or may not agree with the first statement, but it’s hardly in the same category as the second (which is deplorable). It seems to me trivially easy to construct many more such examples.
The hoax paper replaced references to Jews with references to privilege. It’s hardly surprising that rants against an abstract concept are less offensive than anti-Semitic rants.
> “We must fight against the influence of special interests in American politics.”
> “We must fight against the influence of Jews in American politics.”
To be fair, some people do use the former language to mean the special interests of "Jews." No all, but dog whistling does exist and dog whistling is explicitly about covert language. I don't think the special interest groups reference is a great example because it is a common phrase that is used to mean a lot of different things and isn't uncommon in various groups using covert language.
It's a phrase that pretty much every major American politician has used. Of course there are dog whistlers and conspiracy theorists, but I think the less problematic usages predominate. Not that the phrase really means anything as far as I can tell, but that's a separate issue.
I do see what you mean though. There's something inherently gross about juxtaposing two sentences like that. I'm not doing it to suggest any equivalence between them. The point is exactly the opposite (and I would have thought an uncontroversial one): that switching out the major vocabulary items in a sentence can take it from being innocuous to offensive or vice versa.
> Not that the phrase really means anything as far as I can tell, but that's a separate issue.
Actually I think that's explicitly the issue and a main part of my point. In fact, dog whistling or other type of coded language typically depend on ambiguous language. It's literally because language works like an autoencoder. There's what you have in your head that's encoded into what you say and then decoded. The coded language comes through a learned/tuned decoding.
My point is that such phrasing isn't inherently innocuous. Vague language is always inherently dubious. The juxtaposition just makes it more obvious in this case, but it's always true. It may not always be dog whistling but in the least dubious case it lets people fill in whatever they want. That's why I'm saying it isn't a good example.
Hmm. I think it's a fine example because 'special interests' is very rarely used as a dogwhistle for 'Jews'. Can you even point to an example where it is?
I'm not sure why you would be so caught up on "Jews" specifically when there are plenty of other targets that can be used. But yes, that term has been used to refer to Jews the same way one might say "Hollywood types" or "Bankers." I can't think of a specific example, but I think that's a bit high of a bar as I don't have an eidetic memory and it's not like I've been keeping a log of every time some mentions special interest group. Like most people I only remember the broad concepts and general notions. If you're aware of a way to easily Google or search this information, I'd be quite interested to know, because that's a very useful task. In the mean time, I'm sure you could find some if you dug around more around the Ukraine war, with both US right wing and Russian news platforms being the most likely place to find these. There's a lot of anti-semetic conspiracies revolving around Ukraine (with it even being the Russian's main initial propaganda) so that's why I suggest looking there.
Well, we disagree over the usage of 'special interests', and as you say, it's difficult to gather objective data to support or refute either view. It's obviously a vague phrase that could be (and occasionally has been) used to dogwhistle many things, but that doesn't mean that a sentence attacking "special interests" is inherently as offensive as a sentence attacking a specific ethnic or religious group. That is the only point I'm making here, which I suspect you don't even disagree with. Even in the case of a dogwhistle, the whole point of a dogwhistle is that it is less immediately and unambiguously offensive than a straightforward statement of a prejudiced viewpoint.
By the way I'm not 'caught up' on 'Jews'. It's just that 'Jews' is one of the words that the authors of the hoax article replaced.
Do you happen to have the passage in question at hand? Mein Kampf is a fairly long book, and it seems likely to me that there are passages of it which – with key words changed! – would not express anything particularly awful. But it is hard to judge without knowing what the modified passage actually said.
Edit: What the hoaxers actually say suggests that the passages were altered quite substantially: "The last two thirds of this paper is based upon a rewriting of roughly 3600 words of Chapter 12 of Volume 1 of Mein Kampf, by Adolf Hitler, though it diverges significantly from the original.". And you can read the paper here: https://newdiscourses.com/feminist-mein-kampf/#solidarity
Sorry, that was a good faith mistake on my part, and I edited it out before you made this comment. I got confused because the link also states (correctly) that it was rejected from Feminist Theory.
Twice in two comments foldr quoted from a source. Both of those quotes are literally right next to another sentence in the source which is exactly the opposite of what foldr asserted.
The Affila wikipedia page section titled “Grievance studies affair” is four sentences long. The sentence right before foldrs quote, which is the first in the section is this.
>In October 2018, it was revealed that the journal had accepted for publication a hoax article entitled "Our Struggle Is My Struggle: Solidarity Feminism as an Intersectional Reply to Neoliberal and Choice Feminism."
The New Discourses link does say the paper was rejected from a specific journal in the first footnote which is . The second footnote says the article was accepted. Both footnotes quoted here:
> Peer reviewed and rejected by Feminist Theory
> Accepted by Affilia , August 21, 2018
At this point I am no longer sorry for not assuming good faith from someone arguing that a reformulation of kompf could be “not that bad”.
If I was intentionally lying, it seems unlikely that I’d have chosen to link to a page which clearly shows that my statement was false. It was just a careless mistake that I corrected as soon as I noticed it. Relax. I haven’t personally attacked you in this discussion. You’re not making your own point any clearer here by harping on this.
I don’t accept the characterization of being thorough as harping, or that anyone would need to be unrelaxed for that. Is that ironic with me being told to assume good faith?
Rules of thumb by definition don't 'scale' (is this the right word?): they're true on the whole most of the time.
The point of 'assume good faith' is that it's very easy to convince yourself that semi-anonymous people on the internet are liars. Most of the time they're not. It's very easy to inadvertently make factual errors. Published books that have been carefully reviewed are full of them. What chance do random morons ranting on the internet stand?
From what I have read so far, they showed that if you completely fabricate the data and follow the advice of peer reviewers, you can end up at a published paper.
Thank you, Captain Obvious.
Had these valiant warriors heard of Diederik Stapel, they would have known that fraudsters already made this point. Extensively.
These fighters for academic rigour seem to have lost sight of their own academic rigour in this "experiment".
Peer review is insufficient to catch fraudulent submissions. This is not a surprise as much as a misunderstanding of what peer review is.
To be clear: to show something, the authors should have shown that the ideas were obviously false, not that fraudulent data that supports findings would go undetected. Einstein's famous works were considered patently absurd at the time. The only way to substantiate or refute such ideas is with data. An experiment that abuses this to "show" anything is, itself, patently absurd.
Isn't this almost identical to the general admissions process, too?
It seems strange to me that we built institutions on the idea of filtering in/out applicants based on relatively arbitrary criteria, and then express shock/surprise when the reward systems inside that institution are.. basically the same?
There are parallels everywhere, e.g. scientists feeling they must get positive 'groundbreaking discovery!' news reporting about their publications, not just actually doing impactful work, in the same way good grades aren't enough and you need some other impactful story to tell in order to be accepted to many schools.
All of it can be traced back to money, money, money.
> It seems strange to me that we built institutions on the idea of filtering in/out applicants based on relatively arbitrary criteria, and then express shock/surprise when the reward systems inside that institution are.. basically the same?
I think it is impossible to use anything other than noisy signals in many of these processes. I'm not bothered by the noise. But what I am bothered by is pretending the noise doesn't exist or trying to convince people it is a feature and not a bug. Why not just admit the process is noisy and that we're just doing our best? These processes may still be frustrating, but they will be less so and it leaves the door open to fixing issues if they are actually fixable.
With the way we publish science, I find it very odd. We are doing essentially the same thing we have done for centuries with only a few minor tweaks. But unlike the past we don't have many of the same constraints; such as, printing, lack of color, distribution/storage, communication (which is arguably the main purpose of papers), video, and so on. Why are we stuck in the 18th century with a sprinkling of 21st century (plots/graphics and arxiv)? We can track data, reproduction efforts, failures, challenges, and all that stuff. Is not the point of science the pursuit of knowledge? If so, it seems like we're being pretty inefficient. I think there's just something funny about how we publish papers on things like computer graphics and video but constrain our communication methods to a piece of paper.
True for a particular dominant but antiquated and rapidly aging out model of peer review. Peer review as practiced at PeerJ, eLife, F1000, etc. is collaborative, productive, and maintains integrity in a visible way.
Peer review is not inherently terrible. Exploitative rent seeking publishers that commoditize academic careers and outputs, and hold knowledge to ransom, are the problem.
Please, everyone, stop publishing in journals that do harm. Think about what the impact of where you publish a paper is and align your choices with your values.
The peer review system is definitely biased. It's ok for many articles, but there are outliers. For instance really original theories or experimental data. The publication may be stopped to avoid putting the journal's reputation at risk or because the reviewers have personal reasons to not support the article (it would shade their own pet theory, invalidate their research project, they didn't understood it, etc.).
The problem with publishing elsewhere is that many people unable to objectively evaluate the validity of the theory juge its value by the reputation of the journal. Also the other journal will most probably have less visibility. There is thus a higher chance that the other theory will not be reported in reviews.
There are many assumptions in the above comment that I can't agree with. But my experience is with physics, not computer science.
This is a great point about differentiating between filler and seminal papers. And I actually think the word filler is spot on for two reasons.
Firstly, this reminds me of a blog post about what it means to get a PhD [1]. The problem with that view is that the circle of human knowledge is drawn as a solid, while in reality it is very holey. So seminal papers push the border outward, while filler papers fill in the holes. Both valid and necessary, but general public only cares about advancing the borders.
Also, filler papers are needed not only for the sake of completeness, but scientists need papers to demonstrate productivity. E.g. graduate students needs papers on their CV so they can prove to their next employer that they are productive and capable of serious research. So these filler papers also "fill" the role of "filling" up people's resume. The public seem to think that professors publish papers only to announce to the world about their great discoveries, but 90% of the time its filler papers just reporting on business as usual type progress.
The peer review is there to help the journal maintain its reputation by preventing the publication of sub-standard stuff. Period. Sub-standard can mean uninteresting, incomplete, poorly written, or whatever the journal is aiming for. It is not there to safe guard the integrity of the literature against erroneous results...it's purely self-interest on the part of the journal.
In reality, a rejected paper will just be submitted elsewhere until it is eventually accepted. The authors cannot afford to spend 1-2 years worth of work on a project then have nothing to show for it, just because a reviewer didn't "get it". So authors will keep submitting it (hopefully with some improvements based on past reviewer comments, but maybe not) until it "gets through" somewhere, and eventually nearly everything gets the seal of "peer review".
> There is SO much more I could write on this subject, but I'm trying to stay on point (-: