Everyone cites these kind of examples as LLM beating some test or other as some kind of validation. It isn’t .
To me that just tells that the tests are poor, not the LLMs are good. Designing and curating a good test is hard and expensive.
Certifying and examination bodies often use knowledge as a proxy to understanding or reasoning or any critical thinking skills.they just need to filter enough people out, there is no competitive pressure to improve quality at all. Knowledge tests do that just as well and are cheaper.
Standardization is also hard to do correctly, common core is a classic example of how that changes incentives for both teachers and students . Goodhart's law also applies.
To me it is more often than not a function of poor test measurement practices rather than any great skill shown by the LLM.
Passing the CFA or the bar exam while daunting for humans by design does not teach you anything practicing law or accounting. Managing the books of a real company is nothing like what the textbook and exams teaches you .
—-
The best accountants or lawyers etc are not making partner because of their knowledge of the law and tax. They make money same as everyone else - networking and building customer relationships. As long as the certification bodies don’t flood the market they will do well which is what the test does.
There are influencers making many multiples of that for doing far less. Monetary returns has rarely if ever reflected skill, social value, or talent in capitalist economies, this is always been the case, not sure how that is relevant
I was merely commenting on why these tests exist and the dynamics in the measurement industry from an observer, we shouldn't conflate exclusivity or difficulty of a test to its quality or objective.
sure, but if companies find that llm performance on tests is less correlated with actual job performance that human test performance, then that means the test might not be not a useful metric to inform automation decisions
Everyone cites these kind of examples as LLM beating some test or other as some kind of validation. It isn’t .
To me that just tells that the tests are poor, not the LLMs are good. Designing and curating a good test is hard and expensive.
Certifying and examination bodies often use knowledge as a proxy to understanding or reasoning or any critical thinking skills.they just need to filter enough people out, there is no competitive pressure to improve quality at all. Knowledge tests do that just as well and are cheaper.
Standardization is also hard to do correctly, common core is a classic example of how that changes incentives for both teachers and students . Goodhart's law also applies.
To me it is more often than not a function of poor test measurement practices rather than any great skill shown by the LLM.
Passing the CFA or the bar exam while daunting for humans by design does not teach you anything practicing law or accounting. Managing the books of a real company is nothing like what the textbook and exams teaches you .
—-
The best accountants or lawyers etc are not making partner because of their knowledge of the law and tax. They make money same as everyone else - networking and building customer relationships. As long as the certification bodies don’t flood the market they will do well which is what the test does.