Part of the problem is that people seem to want an objective evaluation of a piece of research, and measure the value for taxpayer money. Well, you can’t get an objective evaluation. It’s all subjective. And “impact” as a quantifiable entity in general is nonsense, for one thing the timescales prohibit its measurement at the point where it may be useful.

The solution is to use management. Lots of people object here and say “but nepotism, favouritism” and yep that’s a problem, but it is less of a problem that the decline of western universities. You can circumvent it somewhat by rotation, by involving external figures, by a hierarchy that ensures people are not colluding, but ultimately you just have to trust people and accept some wastage.

People aren’t in academia for the money. It’s a vocation. You’re not going to have many people milking the system. Things went pretty well before the metric culture invaded the academy. They can go well again.

Before introducing the KPIs, a majority of polish science was basically people milking the system and doing barely any (valueable) research. It was seen as an easy, safe and ok paying job where the only major hassle is having to teach the students. You often needed connections to get in. It was partially like that because of the communist legacy, where playing ball with the communist party was the most important merit for promotion, which, over the course of 45 years (the span of communism in Poland), filled the academia management ranks with conformist mediocrities.

Now, after a series of major reforms, there's a ton of KPIs, and people are now doing plenty of makework research to collect the required points, but still little valueable work gets done. Also, people interested in doing genuine science who would be doing it under the old system are now discouraged from joining academia, because in the system they're expected to game the points system and not to do real work.

What is the lesson from this is? Creating institutionalized science is hard? It requires a long tradition and scientific cultural standards and can't be just wished into place by bureaucrats? Also, perhaps it's good to be doing the science for some purpose, which in the US case are often DoD grants, where the military expects some practical application. This application may be extremely distant, vague and uncertain (they fund pure math research!), but still, they're the client and they expect results. Whereas the (unstated) goal of science in Poland seems to be just to increase the prestige of Polish science and its Universities by getting papers into prestigious journals, whereas the actual science being done doesn't matter at all - basically state-level navel gazing.

Relying on altruistic tendencies for people in academia is not adequate. Everyone starts out in academics as school children, and get filtered out or filter themselves out pursuing other things. Those who remain will be the ones who love to learn and teach, those who just cannot accept loss/failure, and, sadly, those who are afraid of change. The more competitive the field becomes, the harder it is to succeed, the more we select for the hyper-competitive or fearful over the altruistic.

The system I was presented with would have meant my supervisors got more say in what I spent my time on than I did. They heavily skewed to supporting existing psychological models of entrepreneurial failure which I wasn't interested in.

The bureaucracy, authoritarianism and endless hoop-jumping around it was a total red flag for me. I opted out.

My conclusions:

PhD grads aren't smarter than others. They're just more willing to put up with bullshit, conform to meaningless rules, and jump through hoops.

Academic research is rarely about the things it says that it's about, and seems to be more about maintaining/improving the career prospects of the academics.

Academic research is heavily affected by political spats with other academics that have nothing to do with the actual subject, but more to do with ego, pride and interpersonal dislike.

Humanity as a whole is losing out on some potentially amazing long term research and researchers because of this dynamic.

I have no idea how to fix it systematically.

PhD grads aren't smarter, they're playing a different game. There's bullshit everywhere, you just chose a different pile to call home -- probably because it smelled better to your reward/bullshit tradeoff.

Why would PhD's choose that pile of bullshit of yours? ...

As you say, "Academic research is rarely about the things it says its about" is a good observation. The reason is that most of the time grants are awarded to professors to try a method (their specialty) to a problem (the thing the grant is about). This looks like the professor is padding his career, but honestly, that's the point.

We would ideally have a huge set of professors with perfect specializations such that a combination of professors could solve any problem. Science funding is ensuring that deep, old expertise is preserved in case it is useful. Grants are a way to simultaneously test that usefulness for modern problems and expand it a little towards modern problems by producing new phd's with slightly mutated expertise. This is why PhDs endure their own type of bullshit, because they want to be part of this particular kind of knowledge legacy. There are other ways of doing this. A problem-first (vs solution-first) approach is kind of better for a different kind of venue, like business, NASA, etc.

Political spats? Absolutely. No contest there. Turf wars are a thing.

You're totally right that academia doesn't have a monopoly on this kind of bullshit. But few other places get away with it so much, and manage to maintain a (rapidly crumbling) reputation of not being 90% bullshit.

I would suggest that your own rapidly crumbling opinion of academia is probably not endemic. The top universities still produce top-tier R&D, and top-tier candidates that go on to do top-tier work for large and small companies.

It's definitely ok and understandable that some (or even many) people would feel differently, but my own opinion (coming from a mid-tier school) is that my training was absolutely of inestimable benefit to my job prospects, and was 100% enabling of my current career in robotics R&D. And I'm guessing that getting admitted to MIT would, without fail, make anyone happy in the entire world. (Though I didn't go to MIT obv)

If you start cheating the metrics, or optimizing a lot towards them, it becomes counter-productive when they change. As such, the most efficient way forward would be to work without trying to optimize for a temporary metric. On the flip side, it would be troublesome to convince people to different, complex forms each time.

What first gave me the idea was the concept of "lubricating" headers (submitting some with random values) for future http protocols, to combat "ossification", where middle boxes start to meddle with them and become obsolete when they don't recognize the new fields, instead of transmitting them.

In that case, if you gamed the "wrong" metrics, you still come out ahead. Not as much as for gaming the right metrics, but still.

Looking at other departments, say, sociology* it's a dumpster. Full of cronies, retired politicians and relatives.

*Just an example which doesn't require "hard" skills like math.

There is only one reasonable way to evaluate research and researchers, that's to evaluate the content of their work and publications by external evaluation panels and tell these panels explicitly that they should not base their assessments solely on indicator counting, but on the overall merits, originality, and prospects of the research according to their subjective opinion. Metrics shouldn't even be used as a tie-breaker, they should only ever be used as weak indicators, and this must be explicitly mentioned in the guidelines.

In addition, you need a few other guidelines and laws. For example, it must be prohibited that someone becomes a postdoc at the same place where they obtained their Ph.D. We have people who study at university X, do their exams at university X, do their Ph.D. at university X under the same professors they always knew, then become postdocs working for their former supervisors and being exploited by them (teaching the same bad seminar their former supervisor teaches the past 20 years), and then get tenure in a rigged call. And the worst thing about it is that they feel entitled to all of this.

You've got to break this vicious cycle, but with your suggestion of using a methodology that worked in 1950s (with an order of magnitude less candidates) this could never be achieved.

I think external evaluation panels (and in particular, from a third country) are the way to go. We already have good examples, for example, in ERC grant panels. The ERC uses exactly the strategy you mention and it has an impeccable reputation, I know many people who applied with or without success but I know no one who felt treated unfairly.

But I'm against blanket rules prohibiting postdocs or positions at the same place of the PhD, at least from my Southern European (Spanish) point of view. This is often touted in Southern European countries because in the US no one does that so it must be evil, so clearly we should ban it to be more like Americans and see if this makes our scientific productivity closer to theirs. But European (especially Southern European) culture is not US culture. People want to settle down, be near their loved ones, and there's nothing intrinsically bad about that that should be punished. Plus, the job market is much less dynamic so even for those who don't mind bumping around, it can be hard to reconcile with a possible spouse who has a career too. And finally, if you push people in this region to move, most of the time the result will be that they end up in a Northern European country (or the US, Canada, etc.) where they make 3x or 4x more, and never come back - once you have experienced a much better salary it's hard to justify returning, I have seen it plenty of times.

Bring on the external evaluation panels, and then there will be no need for any measure forcing people to move, which would reduce inclusivity and thus the talent pool.

But that means that a tenured professor will start acting in non-academically-independent ways at some point before their tenure is up to avoid messing up their re-applicaction.

It is also quite likely that events such as departmental re-orgs could be timed around tenure expiration to eliminate specific job descriptions in order to make re-applying more difficult.

You might be able to achieve the same goals though some combination of making tenure transferrable between cooperating institutions, mandatory sabbaticals, requiring review committees be partly or wholely staffed from outside the institution, etc.

Independent of every other point, this is in my point of view a real problem. It happens so often and I don't understand why there's often no law that at least a postdoc at another institution has to be done.

Not sure what you mean by that, but KPIs are generally put in place by high-level management. Or do you want more micromanagement?

Either way, I think the solution is not more control, quite the opposite. I think the solution is just to remove the extrinsic incentives.

Some people say UBI will cause people to do nothing and that is probably true, but the flip side is that the output of the remaining people will likely be many times higher both in volume and quality (with the total volume much lower but higher quality). Not having their energy completely destroyed by all the busy work necessary to show they are working.

(Speaking about Spain).

The KPIs are there because OFFICIALLY (and this is strictly so) you are not even allowed to get a tenured position without an absurd number of (in Maths) JCR papers IN THE FIRST QUARTILE.

This is so stupid it is not even funny but how can you fight that when your PhD students depend on those metrics?

There was a time in history when tenure made sense, but today the tenure track process forces a lot of people to go after low-hanging fruit that has a high probability of being accepted for publication, instead of trying things that are meaningful to try but may fail.

You are constantly under evaluation whether you have tenure or not.

And what happens if you're poorly evaluated? They can't fire you anyway. Will they move you to a boiler room as a punishment?

That won't happen, mostly because employers have outsourced education and vetting (in the form of requiring bachelor/master degrees) to universities (and the associated costs to governments and/or students who pay tuitions) instead of the old style vocational training/apprenticeship system where the employers had to pay.

Want to restore academia to only those actually interested in science? Make employers pay a substantial tax on jobs requiring academic degrees.

You comment does not seem to contain any explanation as to why 'using management' would solve the problem you allude to. Can you elaborate?

In short, I think it is definitely clear neither the citation nor the journal brand is the best proxy for the worthiness but the system you are proposing is worse while still reliant on subjective judgement.

It's not really about judging people either. When it comes to choosing which people to reward with jobs, promotions, grants, and prizes, we already know the solution: expert panels that spend nontrivial time with each application. Sometimes there are political or administrative reasons that override academic excellence, but in general the academia has figured out how to evaluate shortlisted people.

The real problem is shortlisting people. For example, when a reputable university has an open faculty position, it typically gets tens of great applications. Because the people evaluating the applications are busy professors, they need a way of throwing away most of the applications with minimal effort. From the applicant's perspective, this means you only get a chance if the first impression makes you stand out among other great applicants. And that's why it matters that you publish in prestigious conferences/journals, come from a prestigious university, and have prestigious PhD/postdoc supervisors.

Well, science doesn't care that much for KPIs, per se. It's more that the managers want numbers to steer by.

In academia, getting promoted means more management tasks. So higher up academics have been indoctrinated to want numbers. Is the scourge of management.

As such: not a big fan of your solution.

I agree overall with what you wrote, but have to comment here, because I think this is already not the case in many settings. I can only speak for the US, but in my experience with some other places overseas similar issues are developing.

There are many legitimate hypotheses for why this is the case, but in general at many universities, as far as climbing the academic ladder is concerned, publication metrics are no longer relevant. That is, some baseline is required, but beyond that, most of the focus is on money and grant sizes. I've been in promotion meetings discussing junior faculty that are not publishing and this is brushed aside because they have large grants. I've also repeatedly heard sentiments to the effect of "papers are a dime a dozen, grant dollars distinguish good from bad research."

Again, there's lots of reasonable opinions about this, but I've come to a place where I've decided this is incentivizing corruption. Good research is only weakly correlated with its grant reimbursement, and regardless, it's lead to a focus on something only weakly associated with research quality. Discussions with university staff where you're openly pressured to artificially inflate costs to bring in more indirect funds should raise questions. Just as it's apparent that incentivizing (relatively) superficial bibliometric indices like publication count or h factors leads to superficial science, incentivizing research through grant money has the same effect, but differently.

So yes, going into academics is not the way to make money if that's what you want. However, I think nowadays in the US, it's very much all about the money for large segments, who are milking the system right now at this moment.

Also, in theory, yes, management is the solution, but really management is how we've gotten into this mess. Good management, yes, bad management no. But how do you insure the former?

Fixing this mess academics has slid down (in my perception, maybe everything really is fine) will require a lot of changes that will be controversial and painful to many, and I don't think there's a single magic bullet cure. Eliminating indirect funds is probably one thing, funding research through different mechanisms is another, maybe lotteries, probably opening up grant review processes to the general public. Maybe dissociating scientific research from the university system even more so than has been the case is also necessary. Maybe incentivizing a change in university administration structures. Probably all of the above, plus a lot else.

How to get things to go well again is achievable in theory but how to get there is less clear given the amount of change involved.

Once it's no longer about being in the esteemed and scarce "10%", they won't bother because they don't need to. Imagine a process where the only criteria are technical soundness and novelty, and as long as minimal standards are met, it's a "go". Call it the "ArXiv + quality check" model.

Neither formal acceptance to publish nor citation numbers truly mark scientific excellence; perhaps, winning a "test of time award" does, or appearing in a text book 10 years later.

I've been reviewing occasionally since ~1995, regularly since ~2004, and I've never heard of collusion rings happening in my sub-area of CS (ML, IR, NLP). I have caught people submitting to multiple conferences without disclosing it. Ignoring past work that is relevant is common, more often our of blissful ignorance, and occasionally likely with full intent. I'm not saying I doubt the report, but I suspect the bigger problem that CS has is a large percentage of poor-quality work that couldn't be replicated.

BTW, the most blantant thing I've heard of (from a contact complaining about it on LinkedIn) is someone had their very own core paper from their PhD thesis plagiarised - submitted to another conference (again) but with different author names on it... and they even cited the real author's PhD thesis!

[1] https://ieeexplore.ieee.org/abstract/document/6746236 [2] https://www.youtube.com/watch?v=MFNFScqN47o

I mean I could buy your paper but I would have to know it by heart and understand it in order to defend it.

At least that's how it went when I was studying applied physics (1974-1977).

If I had just bought a paper I would have had a really hard time in the viva voce.

Edit: by -> buy

In my program, many of us would strategize for the 30-60 minutes of the closed door grilling. We sought to give our committee members obvious things to criticize with the PhD student having prepared arguments to defend against these criticisms. E.g., I ashamedly included quite a few spelling and grammar errors in the first few pages of the summary section of the thesis (the only part anyone would actually read) and we spent at least 15 minutes on my horrible writing ability.

In general, the main outcome of the closed door portion of the defense was requests for additional work. It was common for committee members to suggest additional things that could “improve” the thesis work. Not surprisingly, many of these suggestions involved applying a committee member's methods, even if not plausibly applicable, so that one would publish another paper citing the committee member’s work. Some students, including myself, would have a job lined up before the defense to timebox the amount of additional work that could be requested.

"I don't trust your maths" "I don't feel this analysis is right, but I can't describe in what way" "you are clearly not very knowledgeable" and many other similar things.

Asking me a question and then before I can open my mouth answering it yourself, and then insulting me for not answering it was the start of the viva defence and it set the tone for the rest.

I was also heavily criticised for not having cited a paper that came out in-between submitting my thesis and the defence, despite this being literally impossible to have done so, and, despite having already had gotten a job in that time, was given limited time to do additional experiments, write whole new chapters, new code, do new experiments, etc. Ended up adding 90 pages of material to the thesis.

In the end I had to quit the job I had just got, because It would have been impossible to not fail my PhD program and keep the job.

Afterword's, in behind closed doors discussions it was revealed that one guy had pushed for almost all of the required extra work deliberately to try to make me fail, because I had done something he could not.

I mean, personally, I don't really care about reputation of academia, but given that you described a horrible story where somebody basically tried to ruin your career, and given that you decided not to name specific people or institutions, it seems that the whole (UK) academia will have to take the reputation hit for the alleged scandal.

In the first half I was privately grilled by my committee. They wouldn't let the dissertation go to a public defense until they'd satisfied themselves that it was fine.

As far as I can tell the public defenses usually look ceremonial at GMU (at least in its engineering department), but they aren't actually ceremonial. Anyone from the public can ask questions at a public defense, so they aren't ceremonial. However, the goal of the (first) private half was to try to make sure that the defender is ready for arbitrary questions (because he understands the material). So it's unusual for the public to ask questions that the defender isn't able to answer. I got some questions I hadn't heard before in my public defense, but I was able to handle them.

You can see my public defense here: https://www.youtube.com/watch?v=QYH18NpsRu8

So that even if people don't fail their defenses often, there might be many that didn't get theirs scheduled at all? This is just speculation on my part.

More info here: https://michaelnielsen.org/blog/three-myths-about-scientific...

I think the recent wave of low-impact submissions and co-authorship rings is the result of developing countries trying to simplify that process and tying hiring/promotion/pay directly to publication count and related easy metrics.

It may be true that the SNR in research output from developing countries is lower, but there is still lots of good science. But essentially no publishers with good reputation. So even with the same SNR, the increased pool of countries producing science would add to the publication pressure.

One possible issue is that researchers usually need to justify their research to somebody who's not in their field. Conferences are one way to do this. So are citation counts. Both are highly imperfect, but outsiders typically want some signal that doesn't require being an expert in a person's chosen field. The "Arxiv + quality check" model doesn't seem to provide this.

> I suspect the bigger problem that CS has is a large percentage of poor-quality work that couldn't be replicated.

As a sort of ML researcher for several years, I agree.

I don't have data, but from subjective experience, 5-6 years ago most papers in major NLP conferences didn't have an associated code repository. Now, the overwhelming majority do.

There are still many other problems, for example a big one is reporting of spurious improvements that can vanish if you get a less lucky random seed. But at least including code is now common practice.

I suppose the situation regarding common datasets might vary between subfields and NLP tasks, so maybe I just saw a weird corner of it.

Of course the code was also nowhere to be seen.

Availability of code would of course be even more important, both because of replicability and general verifiability, and also because that would allow you to do a comparison with any number of datasets yourself.

Glad to hear that code availability has been improving.

> There are still many other problems, for example a big one is reporting of spurious improvements that can vanish if you get a less lucky random seed.

Considering that a lot of NLP is at least somewhat based on machine learning, don't people do cross-validation or something?

You do a paper showing that problem X can be solved slightly better by downloading and training on a billion tweets.

But you don’t have the copyright to those tweets, so you can’t share data.

> don't people do cross-validation or something

A lot of stable problems comes with a dataset already split into train and test.

That's true. Sometimes you might try to tweak the algorithm itself rather than the data, though, or experiment with different kinds of preprocessing or something, and in those cases it would be helpful to be able to do different experiments with shared datasets.

My limited experiences were from around the time deep learning was only about to become a big thing, so it might have been different then. Maybe you nowadays just throw more tweets and GPUs at the problem.

Also, a "best 10%" conference/journal is valuable -- I have only so much time in the day. There are a few conferences for which it is always a good use of my time to read all the abstracts, plus one or two of the papers that seem most interesting. I can't do that for every conference, or even most conferences, in my area.

So the "best 10%" conference/journal is valuable to the consumers, and the prestiege is valuable to the producers. Therefore I think such a thing would simply re-emerge if you somehow killed it.

Moreover, the correlation with acceptance and impact in existing, but not that high: https://medium.com/ai2-blog/what-open-data-tells-us-about-re...

Of course, only corporate researchers can rely on not publishing in established journals - as their salary and position does not come from "publish or perish" metrics. Ironically, it means that there is more academic freedom in private companies than in academia.

I've been thinking on an off about review system that might improve on things. I'm imagining perhaps: reviewers and authors both get to see who they are, and conflicts of interest can be called out by other people after review and before publication; reviewers are chosen at random, not allowed to bid; reviewers are sent a series of pairs of papers, and asked to choose which one they'd rather see, scores and ultimately publication can be decided by rank choice vote rather than reviewer assignment; comments on paper improvement would be completely optional. Would this be better or worse than what we have? Would it deter explicit collusion?

In my corner of CS, there are plenty reviews of debatable use that are very negative. In fact, I have a pet hypothesis that the average score of the lower scoring but accepted papers is negative (scale from -3 to 3). And I wouldn't be surprised if the median paper's score was negative.

Positive uselessness doesn't seem like that much worse.

I’ve seen a lot of overly and unnecessarily harsh reviews. Anonymizing enables over-stating criticism, it happens routinely. I don’t think I agree that harsh critique is necessary for a healthy review system. It is already the case that good reviews are not extremely harsh, they focus on the facts and are willing to stand by their statements. I don’t personally know that many researchers who have trouble being direct in person and face to face, or of offering constructive criticism.

> As a reviewer you are only advising the editor

This completely depends on the journal or conference. Quite a few of them, especially the larger ones, do not override reviews casually nor often. And what I’m suggesting is a system where this idea can change, where editors can more easily trust the review results, and won’t need to override the decision.

> People are petty.

This might well be true. And so I’m not entirely understanding your argument. It seems to be simultaneously suggesting we have a problem, and defending the status quo as the way it needs to be. What would you suggest as a way to improve the review system so that pettiness has less influence than it does today?

You’re also misunderstanding the point of reviews. They also serve as comments to the authors to modify their manuscript and make it acceptable for publication.

I think this is a little more pessimistic than what the piece says. The NeurIPS (then-NIPS) experiment said that about 60% of papers accepted by one PC got rejected by the other. That doesn't actually mean "the quality of the work has little to do with its odds of acceptance". It may just be that there's a paper has to cross a quality threshold, and once it's past, then the outcome has a lot more variation.

My personal take on NeurIPS specifically is that there's a fraction of bad papers, maybe 40%, that probably shouldn't and won't get in. Then there's a minority of very nice papers that probably should and will get in, maybe 5-10%. And then there are a bunch of middling papers where a lot of it is luck and drawing friendly reviewers. But these aren't bad papers, and you can't really just churn them out, they're just not very good papers.

From memory, it was 25% rejected by both PCs, 15% accepted by both PCs, and the middle 60% random.

As you mention, the problem with having two sets of reviewers is that it's hard enough for conferences like NeurIPS to find one set of qualified reviewers. Usually at least 1/3 of reviewers on any given paper produce a poor review, either because they don't care or because they really lack expertise. Complaining about this is so widespread that it even doubles as a sort of icebreaker for researchers, but nobody has a good solution.

Essentially if you look at review scores for a conference which has say a 40% accept rate (and this is quite similar across fields I'd imagine), you find there's 10-20% (depending on conference) of papers that are clear reject for all reviewers, then there's probably around 10%-15% of papers which are very clear rejects now the rest of the papers are very similar in scores so the cut-off becomes quite arbitrary (and depends on luck as well). This is actually well known for grant applications and a sign that there is likely not enough money in the system.

Maybe it's time to move on from some of these conferences, and focus on interactions that maximize sharing research findings. I know that is unrealistic, but like every other metric, conference acceptance ceases to have value once the metric itself is what people care about.

[1] https://youtu.be/DJFKl_5JTnA?t=853

And one of the suggested solutions (at ~22:00) seems that it would work if it can be adopted - essentially, have top university administrators evaluate top x papers for hiring/tenure decisions and ignore everything beyond that number. What you measure is what you get; if you measure count, you get a deluge of 'least publishable units', if you measure your top 3 papers, then everyone will focus on quality instead of quantity. A counterargument probably is that it's easier for the administors to measure quantity in a way that seems objective and resistant to any arguments or appeals decisions, and it's far harder to objectively compare quality especially if the candidates are from different subfields of research.

The REF is evaluated by humans. I take 2/3 weeks to read a paper. For good papers I might work on them for 6 weeks + to really get into the technique. How can a REF reviewer consume 5 papers to evaluate an academic? How can they consume 200 papers from 40 academics?

The right thing would be to have the department sumbit it's top 5 papers - 3 reviewers could really see what is going on is a department then.

So - how many IEEE conferences are there!? Also things change, it was much easier to get into Neurips 7 years ago... but it's not necessarily the case that the papers from this year will have as much impact or be as good as the papers from then. And as Neurips itself showed, the peer review process is somewhat random - papers are rejected by different panels meaning that getting in is likely an achievement but also possibly a bit lucky. I don't think that evaluating the papers based on where they were published is a good way to allocate public money.

As a strong concrete example, paper 13 from Ferguson's group at Imperial is probably one of the most important documents for the last 20 years (if you live in the UK or France), it was "published" on a website...

It's not even just about academia in general, it's a universal problem caused by lack of personal accountability through the globalization of talent. The exact same dynamic was shown multiple times in HBO's The Wire. Gaming the metrics. LPU's are no different than 10 minute YouTube videos.

When someone is hired (whether for tenure or time-limited), their research as a whole has to be evaluated by external, independent committees who take into account the content of the research and do not base their judgment on indicators only. There is no shortcut around that.

The biggest annoyance nowadays is the decision-makers' insistence on "excellence", though. You cannot have only excellent people everywhere, as per the definition of "excellent", yet this demand is in every fucking guideline for postdocs and tenure-track position. It's absolutely ridiculous.

The problem with this assertion is: non-experts are paying for all of this.

Imagine you are an ordinary taxpayer. You are feeling the pinch yourself, you look around you and see infrastructure crumbling, every day the press says healthcare and this and that is underfunded. Now along comes some scientist, he or she wants a few billion for a new particle collider that will make no difference whatsoever to your life, and their only justification for it is "well other scientists say we should get all this money, and they're cleverer than you, shut up".

Can you see why funding something with no accountability might be considered problematic?

There's also the fact that an expert who can't explain something to a non-expert probably doesn't understand it very well themselves...

> Can you see why funding something with no accountability might be considered problematic?

Maybe you're confusing budget decisions with funding and hiring decisions. These are fundamentally different. Universities and research institutions, as well as national funding authorities, get budgets that are decided politically, i.e., by elected representatives. These can have broad categories and guidelines or preferred research areas (e.g. "excellence initiatives"). Budgets are usually allocated well in advance, for instance our national funding authority gets budget security for 4 year periods (if I'm not mistaken). How they spend it is dictated by political guidelines for the respective period and plenty of complicated national and international laws.

In contrast, I was talking about hiring decisions and decisions about individual funding. How can it not be obvious to you that these decisions need to be made by experts on the basis of CVs and scientific project proposals, not by politicians or other laymen?

It took over 20 years after the Standard Model reached broad acceptance (ie., the experts thought the theory was probably right) for the first supercollider powerful enough to observe the Higgs boson to be financed. This was enough time for policy makers to reach high confidence that the experts had not gone badly wrong and for there to be reasonably well-informed public opinion on the merits of the search.

E.g, for my company's twice-yearly evaluation, everybody writes up a short report on the most impactful stuff they've done including evidence, this is evaluated by their manager to give a score, and then there's a series of group meetings between managers to make sure that the scores are calibrated, including looking at all types of metrics that can be dug up and comparing to our written role descriptions for different levels. It takes a lot of time but creates fair scores.

This is extremely labor intensive, but that's the thing: To create anything resembling fair evaluation of a large group of people that do a large set of different things, you need to do things that are labor intensive. Using a simple set of metrics don't cut it.

Findings and papers from well known individuals (read: twitter accounts) do get far more attention , more citations. Of course, one can argue that, broadly, well known labs and individuals are wel known because of their tendency to do great work, write better papers. And that’s true. However, the above still holds, in my experience as PhD student in ML. Anecdotally, I have seen instances where a less interesting paper from a renowned lab got more attention and eventually more citations than a better paper accepted at the same venue by a less renowned lab on the same topic.

I would also argue that with the increased importance of the (mostly) commercial "high-impact" journals this has become worse. I know that some of the professional (non-expert) editors of these journals specifically look at the citation counts of authors before accepting to send them out to review, because their main aim is to get people reading the articles, not necessarily good science.

I believe journals need to adapt the openreview process. There are obvious parallels in the lifecycle of a journal submission (from submission to acceptance) and the openreview process, except that the latter is accelerated (for better or for worse).

I have to say that I find the review process of the copernicus journals very interesting. You can see a description here: [1]. Unfortunately I don't work in a related field otherwise I would have published there already.

[1] https://www.atmospheric-chemistry-and-physics.net/peer_revie...

> We find considerable evidence that, overall, article citations are positively correlated with tweets about the article, and we find little evidence to suggest that author gender affects the transmission of research in this new media

https://journals.plos.org/plosone/article?id=10.1371/journal...

(I've only skimmed the paper, a few months ago).

- Conferences and participants have increased exponentially over the last few years.

- Students in AI/ML graduate programs have similarly increased.

- Huge numbers of companies are hiring AI/ML graduates.

- What constitutes a true advance AI/ML is difficult to determine. Deep learning is fairly ad-hoc method - a tweak to an existing method that allows you exceed soto (state of the art) is the simplest way to get attention. But that's not a method and so you're many others pushing similar tweaks also.

This environment in particular seems like it would exacerbate all the ordinary pressures to cheat found in the academic environment. It has something of the quality of the last blow-out of a bubble. And the thing is that even with deep learning being real, the dynamics seem fated to push things to the point that expectation are sufficiently far past reality that the whole collapses, for a bit.

Would it help at all if rather than participants reviewing 3 papers, each reviewed 2 papers and validated the review of 3 more papers?

This is computer science here, with things like the set NP whose defining characteristic is that it's easier to check a solution than generate it.

I'm imagining having some standard that reviews are held to in order to make them validatable. When validating a review, you are just confirming that the issues brought up are reasonable. Same for the compliments.

Sure, it's not perfect because the validators wouldn't dive in as deep or have as much context as the reviewers, but sitting here in my obsidian tower of industry, it seems like it would at least make collusion attacks more difficult. Hopefully without increasing the already heavy load on reviewers.

(It very much seems like an incomplete solution -- we only have to look at politics and regulatory capture to see how far wrong things can go, in ways immune to straightforward interventions. Really, you need to tear down as many of the obstacles to a culture of trust as you can. Taping over the holes in a leaking bucket doesn't work for long.)

> In a well-publicized case in 2014, organizers of the Neural Information Processing Systems Conference formed two independent program committees and had 10% of submissions reviewed by both. The result was that almost 60% of papers accepted by one program committee were rejected by the other, suggesting that the fate of many papers is determined by the specifics of the reviewers selected and not just the inherent value of the work itself.

With this much demonstrated discrepancy between two sets of reviewers, it’s hard to believe that adding a validation step would produce a consistent improvement. How can people be expected to find improperly accepted papers when they have less than 50% agreement on the acceptance of good-faith submissions?

Honestly I think this seeming randomness in acceptance is at the heart of why people might think cheating is acceptable. If the process is not reliable, why bother submitting to it?

Suppose that two reviewers independently rank papers 80% on quality and 20% on chance factors. With good odds, the two reviewers will agree with each other on the relative rankings of any given pair of papers. But their lists of the top 10% of papers will largely not be in agreement with each other.

I referenced Kahneham's latest book, Noise, above but this is exactly the problem he focuses on. There are solutions.

Back in grad school a colleague of mine spent nine months on an experiment in a new field and submitted it as a paper to a quality journal. Six months later, the paper was rejected for lack of novelty: One of the reviewers had found a paper with a figure-by-figure duplication of the same experiment -- published on arXiv a week before the rejection decision. Both the managing editor and the author on the arXiv paper were from Chinese universities.

We wrote a rebuttal and submitted a complaint to the journal editor, but no justice was forthcoming. My colleage switched research directions to avoid the collusion and now takes pains not to submit papers without a coauthor who has enough clout in the field to deter blatant research theft. He also avoids dealing with editors from institutions in China.

He ended up graduating two years later than planned.

I wish I had something more substantive to add, but, all I can say is I hope your colleague knows that even if the swindlers and plagiarizers take our research, years of our lives, etc, be they in China or wherever the whole wide world else ain't able to take away the heart of a real one. They know the research ain't theirs, and they know they depend on real producers to be able to commit their crimes, or do anything actually useful and it's not the other way around. These plagiarists are parasites, and one day we'll be free of them.

As long as metrics like impact factor and citation count determine the trajectories of academic careers, plagiarism, collusion, and other forms of academic dishonesty will not go a way.

The fact of the matter is that most academics do not have time to dig into whether someone's work is intellectually dishonest. Being dishonest has a huge payoff as long as you don't cause a scandal.

Anyone who plans to stay in academia should understand that the metagame has changed over the last 50 years. Largely because human attention has not scaled with rate at which academics are exposed to and expected to assimilate new information. There is a larger payoff to exploiting the lack of attention than to earnestly carrying out some meaningful research program. At the very least, do both. By no means do only the latter.

I used to be enamored of academia and its promise. I am now glad there are increasing avenues for success for researchers who choose to leave it, given the limits of their capability to express their ideas honestly with the hope of being recognized for valuable work.

There are metrics and citations being taken, of our dishonest acts. They determine the trajectories of virtually everything relevant to our lives, and can have momentous impacts on the lives of others. I cannot judge any other man or woman, but I know I would be a coward, a fool, and a fraud to diversify any kind of portfolio of mine in this life, by consciously choosing to sprinkle in some exploitation or lies.

I should have been clearer. If you want to stay in academia, not debasing yourself comes at a high cost in terms of your career. It is better to leave than to debase yourself.

Academia is a system worth destroying. Participate in its destruction. Leave. There are so many exciting opportunities to continue doing meaningful work outside of universities and research institutes.

Now there's a system worth destroying.

Some have been posting their papers to pre-print servers and then skipping straight to commercialization attempts. This is especially concerning in the health and fitness world, where some supplement makers and fitness gurus are uploading documents to pre-print servers to give the illusion of being published authors. Casual observers may not be able to tell the difference between published, peer-reviewed papers and some random document uploaded that has a DOI on a pre-print server.

This doesn’t carry much weight in academia, but it can fool non-academic observers. I’m not sure if or how it will translate to CS papers, but I wouldn’t be surprised if skipping peer review becomes more common as the pace of publishing increases.

This is probably a controversial opinion, but I think this makes sense anyway. Peer review & editing from journals made a lot more sense in the world where physically publishing, printing & distributing papers made out of atoms was expensive and difficult. And where retractions and corrections were near impossible, and where we didn't have systems for tracking reputation.

More and more I imagine research becoming like blogging - where "publication" happens by first putting your work online, and then getting feedback in the public domain. And "journals" are replaced by sites like HN or Mastodon which aggregate content and form focal points for a given community.

It won't be perfect, but neither is the current system. And speaking as a mostly independent researcher, the idea of signing over copyright of my work for the privilege of putting my work on their website is preposterous.

Sunlight is the best disinfectant for the kind of corruption described here.

That honestly sounds like it either would be a step backwards, or it would end up re-inventing peer review. I think one of the basic functions of peer review is to provide a basic authoritative quality filter to help make the fire-hose manageable. As someone who lacks the infinite time needed to check everything myself, I find those kinds of filters valuable.

Peer review is good. Pre publication peer review is bad. Bring back the pre WW2 system. End the enormous waste of reviewer time and ludicrous delay. If it was good enough for Einstein it’s good enough. The only person deciding if it’s good enough to publish should be one editor.

IIRC, the current peer review system was created because academic specialization and the quantity of papers increased mid-century to a point where the pre-WWII system became unworkable. Specialization and volume have continued to increase, and it's hard to see how that makes the old system workable again.

We called it Assembl Chronos. It’s now available here at https://provenance.cerebrum.com. Please give it a try, I’d love to hear your thoughts :)

(More info: https://www.prnewswire.com/news-releases/assembl-chronos-a-b...)

I'm trying to understand your story because many details are very different from the things I know about scientific publishing.

Your colleague was a grad student who researched and conducted an experiment in a new field without a supervisor (you say there wasn't a coauthor with enough research cloud)? This never happens in my field and pretty much any technical field I'm aware of.

It then went to the review process and was rejected six month later because a reviewer found (a "copy" of) the paper on arxiv? And the rejection reason was lack of novelty? In many fields arxiv does not count as a "already published" . In particular if the submission date of the article is before the article showing up on arxiv.

Also you say the article on arxiv was very obviously copied. So did this result in plagiarism investigations? While there are many things wrong with the current review process, accusations of instances of plagiarism are typically dealt with very quickly and in my experience pretty much always results in the editors in chief getting involved. So did this happen? Also in my experience there is generally much more scepticism against chinese authors than western authors when this happens. The quality of research and publications from China has dramatically increased in the last 5-10 years though.

I clearly pointed out some gaps in the story that everyone vaguely familiar with scientific publishing would find, that's why I'm asking for clarification.

A supervisor can easily not have "enough clout in the field to deter blatant research theft". If you're a tenured professor from a random university in a random country and are not among the top few names in your field, you probably can and will supervise theses, but good luck convincing an editor from an elite institution to care about you.

And in your last two paragraphs, I think you're being a bit naive... what if the editor in chief is directly involved, or is a friend of the plagiarizer? Whom do you turn to? How do you prove that you did the experiment first if there is no proof but the submission, which is in control of the editor in chief?

I fortunately have never been the victim of such a despicable ploy, but I have seen all sorts of malpractice in journals, including editors that didn't care about blatant plagiarism. I don't think it's a predominant thing, fortunately there are plenty of honest journals and honest editors. But it happens. I'm quite familiar with scientific publishing, having authored a good number of papers, and I find the story perfectly believable.

I don't care how many graduate students you have, there is no way all of that is original research. And in my case, it was an obfuscated version of a very well known theory that any freshman would know.

It's easy to complain about China but that's what the Mainland Chinese system is designed to do. If they're able to break ours then that's just survival of the fittest. Adopt of perish. Who says you have to roll over and simply let them get away with it? Why are our universities collaborating with these "researchers"? Name, shame and blacklist them.

Also, a DMCA request to arXiv would likely work.

> In no case does copyright protection for an original work of authorship extend to any idea, procedure, process, system, method of operation, concept, principle, or discovery, regardless of the form in which it is described, explained, illustrated, or embodied in such work

Could you clarify whether the duplicate paper was also submitted to the same journal or elsewhere, and if submitted whether it was accepted?

I'm trying to figure out whether the point was to steal credit, or to spike your colleague's submission.

> We wrote a rebuttal and submitted a complaint to the journal editor, but no justice was forthcoming.

Yeah this is the sort of thing that seems to only ever get resolved if a public stink is made, often on social media these days, because the integrity of both the editor and the journal is being implicated, which ends up with the can of worms[0] being swept under the rug if at all possible.

Of course people are going to keep tripping over the bump in the rug, and your colleague might not even have been the first victim.

[0] Having to check everything else the editor has ever done for similar misconduct, potentially retractions galore, making victims whole, process improvements, etc.

I wonder if it's worth staring a web-site where friendly scientists put papers side by side and say 'Was this a Rip Off'? Whereby at least the scammers get some form of possible public shaming for as long as the paper exists.

I can just imagine that site coming up in web searches ... how could people not click it?

I have no experience in this area, and I am probably going to ask a dumb question: why not always submit the paper to arXiv before submitting it to a journal to protect against this kind of theft? Wouldn't that clearly and indisputably establish priority?

This is so unfair, some cheaters from China got free research papers to their name and the legit author had to waste two years of their life

One critical vulnerability in the current reviewing pipeline is that the reviewer assignment algorithm places too much weight on the bids. Imagine if you bid on only your friend's paper. The assignment system, if they assign you to any paper at all, is highly likely to assign you to your friend's paper. If you register duplicate accounts or if there are enough colluders, the chance of being assigned to that paper is extremely high.

Fortunately, this is also easy to detect because your bid should reflect your expertise, and in this case it doesn't. What we showed in our paper is that you can reliably remove these abnormal bids. It's not a perfect solution, but it helps.

I was actually very surprised that it is possible to register duplicate accounts at those CSE conferences. We get send a single invite to our work address and need to lock into the system using that email. And we are being nominated to get onto the committee.

I wonder if we could randomly assign reviewers but allow the reviewers to self-report a level of familiarity on the subject matter in general (ideally in advance) and on the paper topic in particular.

"The colluders hide conflicts of interest, then bid to review these papers, sometimes from duplicate accounts, in an attempt to be assigned to these papers as reviewers."

It might also help to attack the ability to create duplicate accounts. Given how relatively few professors exist in the world I'd assume you could put a lot more effort into duplicate account detection than they are right now.

It's, of course, a very hard problem to solve. It takes a lot of effort to evaluate the real impact of research.

"How to get an objective rating in the presence of adversaries"

It is probably extensible to generic reviews as well... so things like the Amazon scam. But in contrast to Amazon, conference participants are motivated to review.

I honestly don't see why all participants can't be considered as part of the peer review pool and everybody votes. I'd guess you run a risk of being scooped but maybe a conference should consist of all papers with the top N being considered worthy of publication. Maybe the remaining could be considered pre-publication... I mean everything is on ArviX anyways.

So instead of bids you have randomization. Kahneman's latest book talks about this and it's been making the rounds on NPR, NyTimes etc...

https://www.amazon.com/Noise-Human-Judgment-Daniel-Kahneman/...

However, they review a limited amount of papers (e.g. 3) - "everybody votes" presumes that everybody has an opinion on the rating of every paper. That does not scale - getting a reasonable opinion about a random paper, i.e. reviewing it, takes significant effort; an event may have 1000 or 10000 papers, having every participant review 3 papers is already a significant amount of work, and getting much more "votes" than that for every paper is impractical.

It's unfeasable and even undesirable for everyone to even skim all the submitted papers in their subfield - one big purpose of peer review is to filter out papers so that everyone else can focus on reading only a smaller selection of best papers instead of sifting through everything submitted. The deluge of papers (even "diarrhea of papers" as called in a lecture linked in another comment) is a real problem, I'm a full-time researcher and I still barely have time to read only a fraction of what's getting written.

This issue reeks with the rank smell of base politics and in/out group dynamics, and humans have been fighting, in the abstract, these issues since the time Egyptians were building the pyramids.

How can there possibly be an "objective rating" when career advancement, peer respect, and big money all are in the mix depending upon results?

In other words objectives are deceiving and rating is based on objectives.

Example: mRNA inventor being sidelined at her university when her method wasn't famous

Example2: Schmidhuber inventing stuff and being forgotten because data and compute were just too small back then

It's all about building a diverse collection of stepping stones. Any new discovery might seem useless and we can't tell which are going to matter years later, but we need the diversity to hedge against the unknown.

Take for example a paper that presents a very innovative method, but with subpar results; and another one that presents an incremental improvement on some existing method, but with results that advance the state of the art. Which is better?

Even if you ask knowledgeable, careful and honest reviewers, you will get contradictory responses, because it's highly subjective whether you rate originality as more important than results or vice versa (and other factors, like whether you think the first method can eventually be improved to be useful or not, which is often just an educated guess). I see this happening all the time, and I don't think it's something that can be "fixed", it's just how humans work.

https://en.m.wikipedia.org/wiki/Collusion

After that event, SIGARCH launched an investigation. After a couple years, here were the results of that investigation.

https://www.sigarch.org/other-announcements/isca-19-joint-in...

Worth noting is that the investigation actually initially found __no__ misconduct. Imagine that? A student kills himself, and you conclude it was the victim's fault, and not the environment that drove him.

It was only until this post [1] emerged that they relaunched the investigation.

[1] https://huixiangvoice.medium.com/evidence-put-doubts-on-the-...

> It should have been unnecessary that we expose these evidence and challenge the result of the investigation, if the committee can drive a responsible, transparent and thorough investigation

An interesting question : Did ACM require the followup Medium article to update their position? I don't know the details of the case. However, merely updating positions when situations are black and white are some of the easiest scenarios. I wouldn't be impressed if black and white situations are assessed as black and white. This doesn't mean that one shouldn't do so. I'd expect those scenarios to be a bare minimum requirement.

Did I miss something? All I can gather from the announcement PDF is that “several individuals” have been disciplined to varying degrees, the least severe being just a warning letter. No names named, no other details. Most of the announcement was just reiterating they took the investigation seriously. Kind of hard to determine from the announcement what details have and have not been considered, and whether certain individuals have been punished too lightly, no?

The announcement does mention a confidential report has been submitted for further review. Did anything concrete ever come out?

(I suppose it would at least be relatively obvious after a while which individuals are subject to a 15-year ban.)

https://omscs.gatech.edu/cs-7641-machine-learning

You almost forget now days that "getting a PhD" or "getting a tenured position" has nothing at all to do with the process of scientific research.

And being landed gentry probably went a long way as well.

As long as you are happy to explore a field that doesn't require millions of dollars of equipment and aren't competing for fame, there's nothing stopping you.

And of course, I could choose to delegate the trust, and to “follow” someone, which would mean to incorporate their rankings, especially in areas where I don’t orient that much.

Do you think this would work?

I do agree that there's probably some cleverer solution on a personal level, but I think the journal system exists as a kind of guard against untrusted actors, and yet fails.

Are you asserting a 'trust' whitelist, a 'distrust' grey or blacklist, or some combination?

Are assertions meant to be linked to or backed by evidence?

Are these assertions publicly visible?

Is there a limit to the number of assertions that can be made?

I think you can see how such a system might devolve into formalized collusion.

*> And of course, I could choose to delegate the trust, and to “follow” someone

In the absence of explicit delegation or following, is trust/distrust intended to be transitive at all (ala PageRank or Advogato WoT)?

I would like to more about this. Can you point to any interviews or other materials that shed light on Burke's thoughts on academia?

This has nothing to do with collusion, but some in this thread are saying that just the number of papers is indicative of a problem.