> Rather than being given questions, contestants are instead given general knowledge clues in the form of answers and they must identify the person, place, thing, or idea that the clue describes, phrasing each response in the form of a question. [0]
Doesn't sound like a test of intelligence to me, so no.
Why? Computers also won chess years ago, but they're not intelligent either? Why is winning a math competition intelligent but a trivia competition or a chess competition not intelligent?
Math and chess are similar in the sense that for humans, both require creativity, logical problem solving, etc.
But they are not at all similar for computers. Chess has a constrained small set of rules and it is pretty straightforward to make a machine that beats humans by brute force computation. Pre-Leela chess programs were just tree search, a hardcoded evaluation function, and lots of pruning heuristics. So those programs are really approaching the game in a fundamentally different way from strong humans, who rely much more on intuition and pattern-recognition rather than calculation. It just turns out the computer approach is actually better than the human one. Sort of like how a car can move faster than a human even though cars don’t do anything much like walking.
Math is not analogous: there’s no obvious algorithm for discovering mathematical proofs or solving difficult problems that could be implemented in a classical, pre-Gen AI computer program.
> there’s no obvious algorithm for discovering mathematical proofs or solving difficult problems that could be implemented in a classical, pre-Gen AI computer program.
Fundamentally opposite. Computer algorithms have been part of math research since they where invented, and mathematical proof algorithms are widespread and excellent.
The llms that are now "intelligent enough to do maths" are just trained to rephrase questions into prolog code.
For the OpenAI case, its unclear. They've not disclosed the method yet. (Though they have previously had an official model that could query wolfram-alpha, so they're not strangers to that method)
But math olympiad questions have been beaten before by AlphaGeometry and a few other's using prolog or similar logic evaluation engines. And it works quite well. (Simply searching LLM prolog gives alot of results on Google and Google scholar)
If openai did it through brute forces text reasoning, its both impressive and frighteningly inefficient.
Even just normal algebra is something llms struggle with, hence using existing algebra solvers is faar more effective.
None of these things are enough by itself. It's rather that they have now solved so many things that the sum total has (arguably) crossed the threshold.
As for solving math problems, that is an important part of recursive self improvement. If it can come up with better algorithms and turn them into code, that will translate into raising it's own intelligence.
I don't wish to join you in framing intelligence as a step function.
I think winning a Go or a chess competition does demonstrate intelligence. And winning a math competition does even more so.
I do not think a trivia competition like Jeopardy demonstrates intelligence much at all, however. Specifically because it reads like it's not about intelligence, but about knowledge: it tests for association and recall, not for performing complex logical transformations.
This isn't to say I consider these completely independent. Most smart people are both knowledgeable and intelligent. It's just that they are distinct dimensions in my opinion.
You wouldn't say something tastes bad because its texture feels weird in your mouth, would you?
I might even think that a symbolic chess program is in some sense more intelligent than a modern LLM. It has a concrete model of the world it operates in along with representation what it can, cannot, and is trying to, do. When LLMs get the right answer, it seems more like... highly-optimized chance, rather than coming from any sort of factual knowledge.
> I think winning a Go or a chess competition does demonstrate intelligence.
Chess is a simple alfa beta pruned minmax seaech tree. If that's intelligent then a drone flight controller or a calculator is aswell.
> association and recall, not for performing complex logical transformations.
By that definition humans doing chess aren't as intelligent as a computer doing chess, since high level chess is heavily reliant on memory and recall of moves and progressions.
I did not share any definitions, only vague opinions. Not that I'd know what it means for a definition to "fall apart".
And the specific bit you cite is barely even a vague opinion; it is my interpretation of the show "Jeopardy!" based on the Wiki article (I've never seen a single episode, wasn't really a thing where I'm from):
> Specifically because it reads like it's about (...) knowledge: it tests for association and recall (...)
Also:
> By that definition humans doing chess aren't as intelligent as a computer doing chess, since high level chess is heavily reliant on memory and recall of moves and progressions.
Yes, I did find this really quite disappointing and disillusioning when I first learned about it. A colleague of mine even straight up quit competitive chess over it.
> it is my interpretation of the show "Jeopardy!" based on the Wiki article
You are spot on though. I mostly wanted to argue that no decent distinction can be made here.
> I did find this really quite disappointing and disillusioning when I first learned about it
ye... same here.
---
I'm personally in the camp that "intelligence" is a human concept. A metric to compare humans. Applying it to computers makes us anthropomorphism computers and think of them as people. Thinking of LLMs as people makes us trust them with bad things.
So we should call them impressive, fluent, fast, useful, good at tasks. Computers already beat us at most math, statistics, searching for information, spacial visualization, information recollection, lossless communication. LLMs just adds to that list, but does nothing new to make the word "intelligent" applicable. Even if we reach the AGI singularity; thinking of them as humans or using human terminology to describe them is a fatal error.
(Destroying earth to make paperclips is arguably the least intelligent thing you could ever do.)
