As a long time LLM enjoyer I just want to mention https://generative.ink/posts/simulators/ as I think it's by far the most insightful take on the GPT LLMs even though it was from before ChatGPT. It's better than blurry jpeg and stochastic parrot etc.
> What pops out of self-supervised predictive training is noticeably not a classical agent. Shortly after GPT-3’s release, David Chalmers lucidly observed that the policy’s relation to agents is like that of a “chameleon” or “engine”:
>> GPT-3 does not look much like an agent. It does not seem to have goals or preferences beyond completing text, for example. It is more like a chameleon that can take the shape of many different agents. Or perhaps it is an engine that can be used under the hood to drive many agents. But it is then perhaps these systems that we should assess for agency, consciousness, and so on.6
Yeah that’s been my thought since the beginning. We have created all sorts of agents to date that are very compelling but lack a bridge into knowledge and synthesis of complex semantics. Wiring an ensemble of the various AI models created over the last 70 years I think is the final step.
I think that article is misleading, because a simulator has rules. An LLM is better thought of as a storyteller, because at best it's going to follow whatever implicit rules there are very loosely and let you make rule changes of your own, more like Calvinball.
Also, whatever loose rules it has are more literary than mathematical. Plot twists often work.
out of distribution tests. if the concept holds consistently over a long period, the concept is the stable thing. If not, then it's only memorizing string densities.
For a number of years we've been basically showing the first to be the case, especially as the model is scaled and the context increases, differentially against the second. String density probabilities can be surprisingly brittle, to be honest. The curse of dimensionality applies to them too, believe it or not, which I believe is why topic discussion, reasoning, and integration over longer distances of text is that differential test that shows pretty clearly that substring memorization/text density stuff is not 'just' what the model is learning. Because mathematically/statistically/from an information density perspective/etc etc otherwise it would be basically impossible, I think.
The essay is long and complicated so I'm not sure how much of it you read closely, but it specifically addresses this distinction between the simulator and the simulacrum.
In the analogy of the essay, your argument would be like saying that reality cannot be simply the application of quantum physics, because you are allowed to make new rules like Calvinball within reality which are different from the rules of quantum physics.
I do understand the difference between a simulator and what's being simulated. I still think they got it all wrong, that the simulator is better called a "writer," the simulated world is better called a "story," and the agent is a better called a "fictional character."
We know there's no deeper level to the simulation/game because we have the entire "game history" (the chat history) and we understand it in approximately same way that the LLM does. (That's what the LLM was trained to do, understand and respond to text the same way we do.) We know that the bot has no hidden state when it's not the bot's turn because of how the bot's API works.
So there's nowhere for a deeper simulation to live. It's as shallow as it looks.
I've always thought of them a bit like improv, since they tend to follow the "yes, and..." rule, by happily continuing whatever direction you want to go. Now that the base models have been fine-tuned to avoid some topics, that's less true than it used to be, but it still feels like the most natural mode of operation.
One thing I've been impressed by recently is ChatGPT's ability to explain epigrams.
How can it explain them effectively if it's just a "stochastic parrot" or a "blurry jpeg"? Neither fits what it does.
Don't such effective explanations require some sort of understanding?
Below is an example explanation, and there are more here: [1]
---
Explain "In software systems it is often the early bird that makes the worm."
This statement refers to the importance of early planning and implementation in software systems. In software development, the early stages of the project, such as requirements gathering, design, and implementation, are crucial for the success of the project.
If the early stages of the project are done well, it can set the foundation for the rest of the development process, making it easier to build and maintain the software system. On the other hand, if these stages are not done well, it can lead to delays, rework, and budget overruns.
The phrase "the early bird that makes the worm" is a reference to the proverb "the early bird catches the worm", which means that those who act early and decisively will have an advantage over others.
Similarly, in software development, if the development team acts early and puts effort into the early stages of the project, they can gain an advantage over competitors, reducing the risk of project failure, and increasing the chances of success.
In summary, the statement "In software systems, it is often the early bird that makes the worm" emphasizes the importance of early planning and implementation in software development, which can set the foundation for the rest of the development process and increase the chances of success.
It literally misses any explanation of the only thing that distinguishes the epigram from the proverb it refers to. While identifying, and reproducing, with simple contextual substitutions, and explanation of the proverb referenced is not a small feat compared to where this kind of technology was a few years ago, its pretty darned far from either (1) actually explaining the epigram, or (2) making the argument you try to make based on explaining the epigram.
That's not even a good explanation for that epigram and the output is similar to many written pieces of content on the internet, which ChatGPT almost surely borrows from via CommonCrawl.
I’d like to know who the author is and what their background/professional experience is. Is that information available? I see they use a handle “moire”.
