> The world of empirical machine learning (ML) strongly relies on benchmarks in order to determine the relative effectiveness of different algorithms and methods. This paper proposes the notion of "a benchmark lottery" that describes the overall fragility of the ML benchmarking process. The benchmark lottery postulates that many factors, other than fundamental algorithmic superiority, may lead to a method being perceived as superior. On multiple benchmark setups that are prevalent in the ML community, we show that the relative performance of algorithms may be altered significantly simply by choosing different benchmark tasks, highlighting the fragility of the current paradigms and potential fallacious interpretation derived from benchmarking ML methods. Given that every benchmark makes a statement about what it perceives to be important, we argue that this might lead to biased progress in the community. We discuss the implications of the observed phenomena and provide recommendations on mitigating them using multiple machine learning domains and communities as use cases, including natural language processing, computer vision, information retrieval, recommender systems, and reinforcement learning.

Edit: By "they", I was actually referring to the linked article. Strangely, even the paper the article is about does not cite "The Benchmark Lottery" at all.

One thing that I'd add (somewhat selfishly as it relates to my PhD work), is the idea of generating datasets that are deliberately challenging for different algorithms. Scale this across a test suite of algorithms, and their relative strengths and weaknesses become clearer. The caveat here is that it requires having a set of measures that quantify different types of problem difficulty, which depending on the task/domain can range from well-defined to near-impossible.

Quantification is fascinating, it seems to be something I take for granted until I actually want to make decisions. It's like I'm constantly trying to forget that analog and digital are two totally separate concepts. I wouldn't really recommend reading Castaneda to anyone but he describes people living comfortably with mutually exclusive ideas in their head walled off by context, and I'd like that sort of understanding.

This impacts the benchmarking process, so I think it's relevant.

Here's the paper that this article is recapping - "Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research", https://openreview.net/forum?id=zNQBIBKJRkd

Abstract: "Benchmark datasets play a central role in the organization of machine learning research. They coordinate researchers around shared research problems and serve as a measure of progress towards shared goals. Despite the foundational role of benchmarking practices in this field, relatively little attention has been paid to the dynamics of benchmark dataset use and reuse, within or across machine learning subcommunities. In this paper, we dig into these dynamics. We study how dataset usage patterns differ across machine learning subcommunities and across time from 2015-2020. We find increasing concentration on fewer and fewer datasets within task communities, significant adoption of datasets from other tasks, and concentration across the field on datasets that have been introduced by researchers situated within a small number of elite institutions. Our results have implications for scientific evaluation, AI ethics, and equity/access within the field."

The reviews seem quite positive, if short. On a skim it looks very solid, offering an empirical look at the dynamics of benchmark usage that IMO seems unprecedented, so I'm not surprised it got positive reviews and accepted.

The word cartel implies collusion. There is no collusion.

Using the word "cartel" in this case is not only wrong, but also... insulting to all the hardworking individuals who have toiled away in obscurity to compile, clean-up, label, package, and publish influential datasets.

Despite the terrible title, the OP is worth a read. It summarizes a recent research paper, also worth a read, about the disproportionate popularity in ML research of a relatively small number of datasets sponsored by and produced at elite institutions (e.g., ImageNet for visual recognition):

https://openreview.net/forum?id=zNQBIBKJRkd

Surely we can find ways to address the winner-take-all dynamics of benchmark popularity without unfairly accusing anyone of running a cartel.

Reviewer 2 is probably the only/best way to address this. More-so if they're a grant reviewer, of course, but paper reviewers will do ok as well.

Are these big "dominant" institutions charging for this data? No, the spend a lot of resources putting them together and give them away free.

Are they preventing others from giving away data? No, but it costs a lot and they bear that cost.

Are they forcing smaller institutions to use their data? No. Its just a free resource they offer.

Do they get the grants themselves because they have some kind of proprietary access? No, the whole point is that the benchmark is open and everyone has access.

So they collect this data, vet it, propose their use for benchmarks and give it away free. What is the complaint? The problem is not even posed as "well, this data is overfitted in papers or we are solving narrower problems". [Edit: they sort of do, but leaving this here so the comment below makes sense as a follow up.]

No the complaint is that these handful of institutions are giving away free data so too many people use it? Can we have more "problems" of this nature?

It's well known that neural networks can easily inherit biases from their training data. It's also well known that datasets generated by western universities are widely used in training and evaluating neural networks.

If my training set is full of pictures of Stanford CS undergraduates, I could end up with a computational photography system that makes everyone look like Stanford CS undergraduates, or a historical photo colourisation system that makes everyone look like Stanford CS undergraduates, or a self driving car pedestrian tracking system that expects 90% of pedestrians to look like Stanford CS undergraduates.

And if the people who make the model say "Hey, our model's biases aren't our responsibility, we're just representing the training data as best we can" and the people who make the training data say "Hey, we never claimed it was perfect, you can take it or leave it" these problems might fall through the cracks.

I don't think this excuse is like the others. If the model doesn't work well because they used bias data, it is the job of the people making the model to find better data (or manipulate the training process to overweight some data and attempt to counteract the bias).

I think the burden of responsibility has to be on the people who make models or put models into products to make sure the model is a good fit for the problem it is solving.

Why then aren't foreign universities/companies simply... building their own datasets?

I mean it does get into that:

"They additionally note that blind adherence to this small number of ‘gold’ datasets encourages researchers to achieve results that are overfitted (i.e. that are dataset-specific and not likely to perform anywhere near as well on real-world data, on new academic or original datasets, or even necessarily on different datasets in the ‘gold standard’)."

The sibling comment on benchmark lottery paper lays this out. But I should modify.

A large institution can make a dataset for X then browbeat other researchers into using X and citing X. Using X also likely leads to citations of derivative work by the lead institution.

- "Nice paper, however the results are not relevant to current research due to the use of X dataset rather than Y or Z datasets score 2/5 do not accept."

- "Nice paper, however the results are of unknown quality due to the use of X dataset 3/5 recommend poster track".

In fact I'd generally say that most paper reviews would drop the first three words of those feedbacks. It's not an unreasonable assertion that progress is measured on standard datasets - but it's also necessary to push back on this.

If you are establishing new baselines using those same models on your non-standard dataset, then one would expect you to put in a good amount of effort to finetune all the knobs to get a reasonable result. If the authors put are able to put in that much effort, then that kind of feedback is definitely unreasonable.

Unfortunately that just adds more work for the reviewer, which is a motive for many reviewers to scrap the paper so they don't have to do the extra work.

That sounds mean, so I will quote (yet again) Geoff Hinton on things that "make the brain hurt":

GH: One big challenge the community faces is that if you want to get a paper published in machine learning now it's got to have a table in it, with all these different data sets across the top, and all these different methods along the side, and your method has to look like the best one. If it doesn’t look like that, it’s hard to get published. I don't think that's encouraging people to think about radically new ideas.

Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad.

https://www.wired.com/story/googles-ai-guru-computers-think-...

Basically a new dataset is like a new idea: it makes the brain hurt, for the overburdened experienced researcher or inexperienced younger researcher alike. Testing a new approach on a new dataset? That makes brain go boom.

Which is a funny state of affairs. Not so long ago it used to be that one sure-fire way to make a significant contribution that would give your paper a leg up over the competition was to create a new dataset. I was advised as much at the start of my PhD (four ish years ago). Seems like this has already changed.

The alternative is either a small dataset that people heavily overfit (eg the MUC6 corpus that was heavily used for coreference at some point where people cared more about getting high numbers than useful results) or things like the Universal Dependencies corpus which are provided by a large consortium of smaller institutions

In fact, as far as I can tell, we are not overfitting in this sense. When I have seen papers examine whether progress on, let’s say, imagenet, actually generalizes to other categorization datasets the answer is yes.

What we have been seeing is that the slope of this graph is flattening out a bit. Whereas in the past a 1% improvement on imagenet would have meant a 1% improvement on a similarly collected dataset, nowadays it will be more like .5% (not exact numbers just using numbers to illustrate what I mean by diminishing returns.)

If an institution or a lab can show that progress on their dataset -better- predicts the progress on a bunch of other closely related tasks, then as researchers become convinced of that, they will switch over. Right now there isn’t a great alternative because it’s not easy to create such a dataset. Scale is critical.

Imagenet really was on the right track as far as collecting images of nearly every semantic concept in the English language. So whatever replaces it will have to be similarly thorough and broad.

In my opinion the biggest weakness of currently existing datasets is that they are typically labeled once per image with no review step. So I think the answer here isn’t

“Let’s get researchers to use smaller datasets from smaller institutions”

It would be more like

“We have to figure out a way to get a bigger, cleaner version of existing datasets and then prove that progress on those datasets is more meaningful”

The realistic way this plays out is that some institution in the “cartel” releases a better dataset and then lots of small labs try it out and show that progress on that dataset better predicts progress in general.

There's a neat set of papers by Recht et al. showing results are slightly overfit to the test partitions of ImageNet and CIFAR-10: rotating examples between the train and test partitions causes systems to perform up to 10-15% worse.

https://arxiv.org/abs/1806.00451 https://arxiv.org/abs/1902.10811

There's another neat bit of work involving MNIST. The original dataset (from the mid-90s) had 60,000 test examples, but the distributed versions that virtually everyone uses has only 10,000 test examples. Performance on these held-out examples is, unsurprisingly, a bit worse:

https://arxiv.org/pdf/1905.10498.pdf

Situation A: Models are slightly overfit to some portions of the test set. But the following holds

IF PerformanceOnBenchmark(Model A) > PerformanceOnBenchmark(Model B) Then PerformanceOnSimilarDaset(Model A) > PerformanceOnSimilarDataset(Model B)

Therefore progress on the benchmark is predictive of progress in general.

Situation B: The relation does not hold, and therefore progress on the benchmark does not predict general progress. This almost always happen if you train a deep neural network long enough: train performance goes up, but test performance goes down.

If you look at figure 2 of the first paper you sent, you will note that it shows we are in situation A and not situation B.

Situation A overfitting = diminishing returns on improvements on benchmark, but the benchmark is still useful. Situation B overfitting = the benchmark is now useless

This is a problem that is more common for classification problems, I think. Generative and self-supervised models (trained with augmentation) tend to just get better forever (with some asymptote) because memorization isn't a viable strategy.

I personally think image classification is mostly a silly problem to judge new algos on as a result, and leads to all kinds of nonsense as people try to extrapolate meaning from new results.

As it turned out, differences between datasets proved significantly larger (by a big margin) than differences between algorithms. And the most popular datasets in fact included biases and eccentricities that were bound to cause such problems.

https://bastibe.github.io/Dissertation-Website/ (figures 12.9-12.11, if you're interested)

The pay2publish-paper was sent back by 2 journals with reviews indicating exactly "our method is better, just use the right (our) thing" (which I _all_ refuted for the professor by doing it, but the editor wouldn't hear anymore...). And then there's papers predicting features through a complex preprocessing pipeline in these journals. Academia and big companies are just idiotic.

Oh well. I'm glad I got my degree, and could leave academia relatively unharmed.

Now I work in AI (engineering!), where most science is highschool-level "hey, I tried stuff, and things happened. Dunno why, though". It's just ridiculous.

The problem here is, though, most publishers don't accept papers if the new proposal isn't backed by benchmarks by the well-known datasets. Even though it can be a competitive approach for a specific field, they reject the paper anyway if it's not performed well on the datasets.

There is a good reason to reject if you are not using a standard dataset. How can you compare the results of two approaches, to say natural language inference of whether one sentence entails another, without results being tested on the same dataset?

I think one thing overlooked in the conversation is that many papers start with a standard baseline and then use another dataset to establish additional results.

In my experience in nlp also, journals and conferences tend to establish datasets of their own when they make a call for submission. Often these are called the "shared task" track. ACL has operated this way for decades.

Granted, datasets have grown larger and larger over time. This concentration is actually a healthy sign of a community maturing and converging towards common benchmarks.

These datasets are open to all for research and have fueled considerable progress both in academia and industry.

The authors would be well-advised to look at what is happening in the community. Excerpts from the program of NeurIPS happening literally this week:

Panels: - The Consequences of Massive Scaling in Machine Learning - The Role of Benchmarks in the Scientific Progress of Machine Learning - How Copyright Shapes Your Datasets and What To Do About It - How Should a Machine Learning Researcher Think About AI Ethics?

All run by top-notch people coming from both academia and industry, and from a variety of places in the world.

I am not saying that everything is perfect, but this article paints a much darker picture than needed.

For example, if there are 50 datasets of historical weather data how can I determine which one is garbage?

Any more papers on the subject that you can recommend?

Now we have exploded that corpus to include everything anyone says online, and we are worried that the lack of curation means we cannot be sure what the models will come back with.

Its rather like sending our kids out to find an education themselves, and finding three come back with nothing, two spent years learning from the cesspit of extremism, two are drug addicts hitting "more" and one stumbled into a library.

Just a thought but I dont think journalism is really about writing newspaper articles. It really is about curating the whole wide world and coming back with "this is what you need to know". Journalism is the curation AI needs...

'A fact beats theory every time...And I think economists have focused on theory because it's easier than collecting data.'

When I read this article this is exactly what I thought of -- modeling/researchers always focus on the low hanging fruit which is sitting in their comfy chair in their ~200 year old university developing hyper-complicated models rather than going out and collecting data that would answer their questions.

I'll have to read the paper to see what exactly it says on this but my knowledge of NLP benchmark datasets is exactly the opposite: the majority are simply no use for measuring the capabilities they're supposed to be measuring. For example, natural language _understanding_ datasets are typically created as multiple-choice questionnaires, so that, a) there is already a baseline accuracy that a system can "hit" just by chance (but which is almost never noted, or compared against, in papers) and b) a system that's good at classification can beat the benchmarks black and blue without doing any "understanding". And this is, indeed, what's been going on for quite a while now with large language models that take all the trophies and are still dumb as bricks.

To make matters worse, NLP also doesn't have any good metrcis of performance. Stuff like the BLEU scores are just laughably inadequate. NLP is all bad metrics over bad benchmarks. And NLP results are much harder to just "eyball" than machine vision results (and the models are much harder to interpret than machine vision models where you can at least visualise the activations and see... something). I think NLP is much, much worse than machine vision.

https://paperswithcode.com/dataset/debatesum

https://huggingface.co/datasets/Hellisotherpeople/DebateSum

https://scholar.google.com/citations?user=uHozRV4AAAAJ&hl=en

Minor typo in this article...

ARTICLE: Among their findings – based on core data from the Facebook-led community project Papers With Code (PWC) – the authors contend that ‘widely-used datasets are introduced by only a handful of elite institutions’, and that this ‘consolidation’ has increased to 80% in recent years.

...but right after, they quote the paper and clearly it is 50% not 80%. See the quote from the paper:

PAPER: ‘[We] find that there is increasing inequality in dataset usage globally, and that more than 50% of all dataset usages in our sample of 43,140 corresponded to datasets introduced by twelve elite, primarily Western, institutions.’

...and the article is leaving out this relevant quote from the paper:

PAPER: Moreover, this concentration on elite institutions as measured through Gini has increased to over 0.80 in recent years (Figure 3 right red). This trend is also observed in Gini concentration on datasets in PWC more generally (Figure 3 right black).

...and in general the article is right that inequality is increasing over time, but Gini is a specific metric to measure inequality, and 0.80 is not the same as 80% inequality.

Edit: After asking a friend seems most of their research is with Chinese official research institutes and Sensetime. It makes sense now.

[1] Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks, https://arxiv.org/pdf/2103.14749.pdf

Once solved, these no longer are research areas. Getting X score on a Y validation set means you get your s/w drivers license.

For a computer science analogy: It is a paper with the finding that most successful computer languages are created at prestigious institutes. An obvious -not a bad - finding. Not like you could give the motivation, skills, expertise, resources, and time to a small new institute, and expect these to come up with a new language which the community will adopt.

Yes, if you write and publish a good data set, and it gets adopted by the community, then you gain lots of citations. This reward is known, and therefore some researchers expend the effort of gathering and curating all this data.

It is not a "vehicle for inequality in science". Benchmarks in ML are a way to create an equal playing field for all, and allows one to compare results. Picking a non-standard new benchmark to evaluate your algorithm is bad practice. And benchmarks are the true meritocracy. Beat the benchmark, and you too can publish. No matter the PR or extra resources from big labs. It is test evaluation that counts, and this makes it fair. Other fields may have authorities writing papers without even an evaluation. That's not a good position for a field to be in.

> The prima facie scientific validity granted by SOTA benchmarking is generically confounded with the social credibility researchers obtain by showing they can compete on a widely recognized dataset

Here, authors pretend social credibility of researchers has any sway. There is no social credibility for a Master's student in Bangladesh, but when they show they can compete, then they can join and publish. Wonderful!

Where the authors use the long history of train-test splits, to pose the cons have outweighed the benefits, they should reason more and provide more data to actually show this and get the field to get along. Ironically, people take more note of this very paper, due to the institution affiliation of the authors. I do too. If they had a benchmark, I would have first looked at that.

> Given the observed high concentration of research on a small number of benchmark datasets, we believe diversifying forms of evaluation is especially important to avoid overfitting to existing datasets and misrepresenting progress in the field.

I believe these authors find diversity important. But for overfitting, these should look at actual (meta-) studies and data. This seems conflicting. For instance:

> A Meta-Analysis of Overfitting in Machine Learning (2019)

> We conduct the first large meta-analysis of overfitting due to test set reuse in the machine learning community. Our analysis is based on over one hundred machine learning competitions hosted on the Kaggle platform over the course of several years. In each competition, numerous practitioners repeatedly evaluated their progress against a holdout set that forms the basis of a public ranking available throughout the competition. Performance on a separate test set used only once determined the final ranking. By systematically comparing the public ranking with the final ranking, we assess how much participants adapted to the holdout set over the course of a competition. Our study shows, somewhat surprisingly, little evidence of substantial overfitting. These findings speak to the robustness of the holdout method across different data domains, loss functions, model classes, and human analysts.

Oh boy. PWC is not even close to a representative sample of what datasets are being used in papers. It's also often out of date.

Most stuff is proven by using advanced pattern matching as "intuition", and then breaking problems down into things we can pattern match and prove.

I'm not sure what conceptual understanding you think isn't pattern matching. They are just rules about patterns and interactions of patterns. These rules were developed through pattern matching.

> A combination of independent business organizations formed to regulate production, pricing, and marketing of goods by the members.

And it does seem to apply ¯\_(ツ)_/¯