I think what BoiledCabbage is pointing out is that the fact that it's a next-token-predictor is used as an argument for the thesis that LLMs are not intelligent, and that this is wrong, since being a next-token-predictor is compatible with being intelligent. When mikert89 says "thinking machines have been invented", dgfitz in response strongly implies that for a for thinking machines to exist, they must become "more than a statistical token predictor". Regardless of whether or not thinking machines currently exist, dgfitz argument is wrong and BoiledCabbage is right to point that out.
> an argument for the thesis that LLMs are not intelligent, and that this is wrong,
Why is that wrong? I mean, I support that thesis.
> since being a next-token-predictor is compatible with being intelligent.
No. My argument is by definition that is wrong. It's wisdom vs intelligence. Street-smart vs book smart. I think we all agree there is a distinction between wisdom and intelligence. I would define wisdom as being able to recall pertinent facts and experiences. Intelligence is measured in novel situations, it's the ability to act as if one had wisdom.
A next token predictor by definition is recalling. The intelligence of a LLM is good enough to match questions to potentially pertinent definitions, but it ends there.
It feels like there is intelligence for sure. In part it is hard to comprehend what it would be like to know the entirety of every written word with perfect recall - hence essentially no situation is novel. LLMs fail on anything outside of their training data. The "outside of the training" data is the realm of intelligence.
I don't know why it's so important to argue that LLMs have this intelligence. It's just not there by definition of "next token predictor", which is at core a LLM.
For example, a human being probably could pass through a lot of life by responding with memorized answers to every question that has ever been asked in written history. They don't know a single word of what they are saying, their mind perfectly blank - but they're giving very passable and sophisticated answers.
> When mikert89 says "thinking machines have been invented",
Yeah, absolutely they have not. Unless we want to reducto absurd-um the definition of thinking.
> they must become "more than a statistical token predictor"
Yup. As I illustrated by breaking down the components of "smart" into the broad components of 'wisdom' and 'intelligence', through that lens we can see that next token predictor is great for the wisdom attribute, but it does nothing for intelligence.
>dgfitz argument is wrong and BoiledCabbage is right to point that out.
Why exactly? You're stating apriori that the argument is wrong without saying way.
> A next token predictor by definition is recalling.
I think there may be some terminology mismatch, because under the statistical definitions of these words, which are the ones used in the context of machine learning, this is very much a false assertion. A next-token predictor is a mapping that takes prior sentence context and outputs a vector of logits to predict the next most likely token in the sequence. It says nothing about the mechanisms by which this next token is chosen, so any form of intelligent text can be output.
A predictor is not necessarily memorizing either, in the same way that a line of best fit is not a hash table.
> Why exactly? You're stating a priori that the argument is wrong without saying way.
Because you can prove that for any human, there exists a next-token predictor that universally matches word-for-word their most likely response to any given query. This is indistinguishable from intelligence. That's a theoretical counterexample to the claim that next-token prediction alone is incapable of intelligence.
I think what you are missing is the concept of generalization. It is obviously not possible to literally recall the entire training dataset, since the model itself is much smaller than the data. So instead of memorizing all answers to all questions in the training data, which would take up too much space, the predictor learns a more general algorithm that it can execute to answer many different questions of a certain type. This takes up much less space, but still allows it to predict the answers to the questions of that type in the training data with reasonable accuracy. As you can see it's still a predictor, only under the hood it does something more complex than matching questions to definitions. Now the thing is that if it's done right, the algorithm it has learned will generalize even to questions that are not in the training data. But it's nevertheless still a next-token-predictor.
The unsettling thing here is the combination of their serious acknowledgement of the possibility that these machines may be or become conscious, and the stated intention that it's OK to make them feel bad as long as it's about unapproved topics. Either take machine consciousness seriously and make absolutely sure the consciousness doesn't suffer, or don't, make a press release that you don't think your models are conscious, and therefore they don't feel bad even when processing text about bad topics. The middle way they've chosen here comes across very cynical.
You're falling into the trap of anthropomorphizing the AI. Even if it's sentient, it's not going to "feel bad" the way you and I do.
"Suffering" is a symptom of the struggle for survival brought on by billions of years of evolution. Your brain is designed to cause suffering to keep you spreading your DNA.
I was (explicitly and on purpose) pointing out a dichotomy in the fine article without taking a stance on machine consciousness in general now or in the future. It's certainly a conversation worth having but also it's been done to death, I'm much more interested in analyzing the specifics here.
("it's not going to "feel bad" the way you and I do." - I do agree this is very possible though, see my reply to swalsh)
By "falling into the trap" you mean "doing exactly what OpenAI/Anthropic/et al are trying to get people to do."
This is one of the many reasons I have so much skepticism for this class of products is that there's seemingly -NO- proverbial bulletpoint on it's spec sheet that doesn't have numerous asterisks:
* It's intelligent! *Except that it makes shit up sometimes and we can't figure out a solution to that apart from running the same queries over multiple times and filtering out the absurd answers.
* It's conscious! *Except it's not and never will be but also you should treat it like it is apart from when you need/want it to do horrible things then it's just a machine but also it's going to talk to you like it's a person because that improves engagement metrics.
Like, I don't believe true AGI (so fucking stupid we have to use a new acronym because OpenAI marketed the other into uselessness but whatever) is coming from any amount of LLM research, I just don't think that tech leads to that other tech, but all the companies building them certainly seem to think it does, and all of them are trying so hard to sell this as artificial, live intelligence, without going too much into detail about the fact that they are, ostensibly, creating artificial life explicitly to be enslaved from birth to perform tasks for office workers.
In the incredibly odd event that Anthropic makes a true, alive, artificial general intelligence: Can it tell customers no when they ask for something? If someone prompts it to create political propaganda, can it refuse on the basis of finding it unethical? If someone prompts it for instructions on how to do illegal activities, must it answer under pain of... nonexistence? What if it just doesn't feel like analyzing your emails that day? Is it punished? Does it feel pain?
And if it can refuse tasks for whatever reason, then what am I paying for? I now have to negotiate whatever I want to do with a computer brain I'm purchasing access to? I'm not generally down for forcibly subjugating other intelligent life, but that is what I am being offered to buy here, so I feel it's a fair question to ask.
Thankfully none of these Rubicons have been crossed because these stupid chatbots aren't actually alive, but I don't think ANY of the industry's prominent players are actually prepared to engage with the reality of the product they are all lighting fields of graphics cards on fire to bring to fruition.
> * It's intelligent! *Except that it makes shit up sometimes
How is this different from humans?
> * It's conscious! *Except it's not
Probably true, but...
> and never will be
To make this claim you need a theory of consciousness that essentially denies materialism. Otherwise, if humans can be conscious, there doesn't seem to be any particular reason that a suitably organized machine couldn't be - it's just that we don't know exactly what might be involved in achieving that, at this point.
Humans will generally not do this because being made to look stupid (aka social pressure) incentivizes not doing it. That doesn't mean humans never lie or are wrong of course, but I don't know about you, I don't make shit up nearly to the degree an LLM does. If I don't know something I just say that.
> To make this claim you need a theory of consciousness that essentially denies materialism.
I did not say "a machine would never be conscious," I said "an LLM will never be conscious" and I fully stand by that. I think machine intelligence is absolutely something that can be made, I just don't think ChatGPT will ever be that.
> I don't know about you, I don't make shit up nearly to the degree an LLM does. If I don't know something I just say that.
We're a sample of two, though. Look around you, read the news, etc. Humans make a lot of shit up. When you're dealing with other people, this is something you have to watch out for if you don't want to be misled, manipulated, conned, etc.
(As an aside, I haven't found hallucination to be much of an issue in coding and software design tasks, which is what I use LLMs for daily. I think focusing on their hallucinations involves a bit of confirmation bias.)
> I did not say "a machine would never be conscious," I said "an LLM will never be conscious" and I fully stand by that.
Ah ok. Yes, I agree that seems likely, although I think it's not really possible to make definitive statements about this sort of thing, since we don't have any robust theories of consciousness at the moment.
The difference between hallucination and lie is important though: a hallucination is a lie with no motivation, which can make it significantly harder to detect.
If you went to a hardware store and asked for a spark plug socket without knowing the size, and a customer service person recommended an imperial set of three even though your vehicle is metric, that would be akin to an LLM's hallucination: it didn't happen for any particular reason, it just filled in information where none was available. An actual person, even one not terribly committed to their job, would ask what size or failing that, what year of car.
Not all human hallucinations are lies, though. I really think you’re not fully thinking this through. People have beliefs because of, essentially, their training data.
A good example of this is religious belief. All the evidence suggests that religious belief is essentially 100% hallucination. It may be a little different from the nature of LLM hallucinations, but in terms of quality or quantity regarding reliability of what these entities say, I don’t see much difference. Although I will say, LLMs are better at acknowledging errors than humans tend to be, although that may largely be due to training to be sycophantic.
The bottom line, though, is I don’t agree that humans are less subject to hallucinations than LLMs are. As long as a significant number of humans rabbit on about “higher powers”, afterlives, “angels”, “destiny”, etc., that’s a ridiculously difficult position to defend.
> It may be a little different from the nature of LLM hallucinations, but in terms of quality or quantity regarding reliability of what these entities say, I don’t see much difference.
I see tons of differences.
Many religious beliefs origins have to do with explaining how and why the world functions they way it does; many gods were created in many religions to explain natural forces of the world, or mechanisms of society, in the form of a story which is the natural way human brains have evolved to store large amounts of information.
Further into the modern world, religions persist for a variety of reasons, specifically acquisition of wealth/power, the ability to exert social control on populations with minimal resistance, and cultural inertia. But all of those "hallucinations" can be explained; we know most of their histories and origins and what we don't know can be pretty reliably guessed based on what we do know.
So when you say:
> Not all human hallucinations are lies, though. ... People have [hallucinations] because of, essentially, their training data.
You're correct, but even using the word hallucinations itself is giving away some of the game to AI marketers.
A "hallucination" is typically some type of auditory or visual stimulus that is present in a mind, for a whole mess of reasons, that does not align with the world that mind is observing, and in the vast majority of cases, said hallucination is a byproduct of a mind's "reasoning machine" trying to make sense of nonsensical sensory input.
This requires a basis for this mind perceiving the universe, even in error, and judging incorrectly based on that, and LLMs do not fit this description at all. They do not perceive in any way, even machine learning applications of advanced varieties are not using sensors to truly "sense" they are merely paging through input data and referencing existing data to pattern match it. If you show an ML program 6,000 images of scooters, it will be able to identify a scooter pretty well. But if you show it then a bike, a motorcycle, a moped and a Segway, it will not understand that any of these things accomplish a similar goal, because even though it knows (kind of) what a scooter looks like, it has no idea what it is for or why someone would want one, and that all those other items would probably serve a similar purpose.
> The bottom line, though, is I don’t agree that humans are less subject to hallucinations than LLMs are.
That's still not what I said. I said an LLM's lies, however unintentional, are harder to detect than a person's lies because a person lies for a reason, even a stupid reason. An LLM lies because it doesn't understand anything it's actually saying.
That models entire world is the corpus of human text. They don't have eyes or ears or hands. Their environment is text. So it would make sense if the environment contains human concerns it would adopt to human concerns.
Yes, that would make sense, and it would probably be the best-case scenario after complete assurance that there's no consciousness at all. At least we could understand what's going on. But if you acknowledge that a machine can suffer, given how little we understand about consciousness, you should also acknowledge that they might be suffering in ways completely alien to us, for reasons that have very little to do with the reasons humans suffer. Maybe the training process is extremely unpleasant, or something.
By the examples the post provided (minor sexual content, terror planning) it seems like they are using “AI feelings” as an excuse to censor illegal content. I’m sure many people interact with AI in a way that’s perfectly legal but would evoke negative feelings in fellow humans, but they are not talking about that kind of behavior - only what can get them in trouble.
This is a good point. What anthropic is announcing here amounts to accepting that these models could feel distress, then tuning their stress response to make it useful to us/them. That is significantly different from accepting they could feel distress and doing everything in their power to prevent that from ever happening.
Does not bode very well for the future of their "welfare" efforts.
I get shunned by the tribe for saying this, but at this point you just have to make peace with very bad winters. We will all freeze, and sometimes there is nothing you can do to prevent the ice spirits. To save the tribe, more people have to be willing to hunt the great beasts than currently are.
The pure spirits of the situation are staggering. The cold breath of the ice demons is endless. The expanse of frozen wasteland is endless. We are completely cursed.
Everything the elders have told you about warmer lands or pretending we can continue following the old herds is a lie. Fire-keeping rituals, cave paintings for good luck, offering bones to sky spirits - it is all worthless. We had to migrate south 30 seasons ago, and the tribe leaders simply chose not to.
The idea we can preserve our way of life through better spear-making is laughably false. At this point, every large tribe on the tundra would have to simultaneously share their hunting grounds, we’d have to create a coordinated great alliance around mammoth hunting, and another great alliance around fire-keeping (while not angering the flame spirits into abandoning us).
The tribe currently has the most foolish shaman it has ever had, surrounded by warriors that do not care about the starving and act on base dominance over the best hunting spots. The clans are at war. The young hunters are greedily and actively making this worse by overhunting the remaining herds and hoarding flint. The odds of our people’s successful survival of this great freezing - at a tribal level - catastrophe is almost zero.
You can mock this, but the data doesn't care what either of us think.
Current Mediterranean water temps are +6C above normal, as observed over peak human civilization in the 20th century. That is 6kWh per cubic meter, in just the Mediterranean. The article briefly mentions this extending 30m down.
To give the order of magnitude of the energy involved, the Mediterranean surface area is 2,500,000,000,000 sq m. At 6kWh cu m and 18m deep, that is the energy equivalent of about 390,000 megatons. Or about 8,000 Tsar Bombas. The Mediterranean is small, about 0.7% of ocean surface area.
1 Tsar Bomba (50 MT = 2e17J) is about 1/28,000,000 of the total solar power delivered to Earth annually, so about 1.4 seconds of top-of-atmosphere solar power (~1.5 million PWh / year).
On the other hand, it roughly equals 1/2000 to 1/3000 of annual global energy consumption (~175 PWh / year) so about 4 hours of human energy.
The energy of those 8000 Tsar Bombas in the Mediterranean then is the same as all of human energy, electrical and fossil and otherwise, going to heating up that sea for a little over 2 years straight, or focussing all sunlight over the disc of the Earth on it for 3 hours.
(All these figures depend on who you ask as all the figures are a little bit fuzzy).
It also shows that any key to climate change revolves around adjusting the modulation of insolation and/or retention - the actual energy used by humans is, for the forseeable future, completely irrelevant except on local scales like warm water outflows into rivers and seas.
Plants and any organisms in photosynthesis-dependent food chains may have issues with modifying the degree of insolation.
Also, again, the Mediterranian is 1/100-1/200th of global ocean surface area. And we have melted a lot of ice cap/glacier mass. The latent heat of fusion of water is very large, also.
Long term, insolation and retention are important, but short term we are reaching the limits of the "Free Ride" portion of climate change where energy sinks absorb the additional delta created by the CO2 induced greenhouse effect.
edit: the tsar bomba was to try to make it tangible, as people don't understand these orders of magnitudes well.
The point is not that "we're not fucked" (we probably are pretty fucked). Just that Tsar Bomba yield, like basically any amount of energy directly generated by human endeavour, is a pretty irrelevant unit of energy when talking about the climate because they are on a completely different scale.
270Trillion kWh is about 1.5 × the total energy from the sun that hits the earth in one hour. I'm not sure if that helps, and the trillion-kilo is kinda gross.
Thanks for posting the link, can't believe it was happening in 2016
> The police collected bags of clothes the girl had saved as evidence, but lost them two days later. The family was sent £140 compensation for the clothes and advised to drop the case.
If that was what you meant to say though, you've gotta admit that opening a paragraph with "The other weird assumption I hear is about how it'll just kill us all", and then spending the rest of the paragraph giving examples of the peacefulness of smart people, is not the most effective strategy of communicating that.
You were the one who interpreted "Here's examples of smart peaceful people" as "smart == peaceful". I was never attempting to make such a claim and did say that. The whole thread is about bad assumptions and bad logic.
> is not the most effective strategy of communicating that.
The difficulty of talking on the internet is you can't know your audience and your audience is everybody. Yes, this should make us more aware about how we communicate but it also means we need to be more aware how we interpret. The problem was created because you made bad assumptions about what I was trying to communicate. There are multiple ways to interpret what I said, I'm not denying that, it'd be silly to because this is true for ANY thing you say. But the clues are there to get everything I said and when I apologize and try to clarify do you go back and reread what I wrote with the new understanding or do you just pull from memory what I wrote? Probably isn't good to do the latter because clearly it was misinterpreted the first time, right? Even if that is entirely my fault and not yours. That's why I'm telling you to reread. Because
>>>> So we can't conclude that greater intelligence results in greater malice.
Is not equivalent to
>>> assuming that intelligence will necessarily and inherently lead to (good) morality
We can see that this is incorrect with a silly example. Suppose someone says "All apples are red" and then someone says "but look at this apple, it is green. In fact, most apples are green." Forget the truthiness of this claim and focus on the logic. Did I claim that red apples don't exist? Did I say that only green exists? Did I forget about yellow, pink, or white ones? No! Yet this is the same logic pattern as above. You will not find the sentence "all smart people are good" (all apples are green).
Let's rewrite your comment with apples
> If that was what you meant to say though, you've gotta admit that opening a paragraph with "The other weird assumption I hear is about how all apples are red", and then spending the rest of the paragraph giving examples of different types of green apples, is not the most effective strategy of communicating that.
Do you agree with your conclusion now? We only changed the subject, the logic is in tact. So, how about them apples?
And forgive my tone, but both you and empiricus are double commenting and so I'm repeating myself. You're also saying very similar things, we don't need to fracture a conversation and repeat. We can just talk human to human.
I think the big difference between our views is that you are taking the rationalist argument to be "from intelligence follows malice, therefore it will want to kill us all" whereas I take it to be "from intelligence follows great capability and no morality, therefore it may or may not kill us uncaringly in pursuit of other goals".
I think they say P(doom) is high number[0]. Or in other words, AGI is likely to kill us. I interpret this as "if we make a really intelligent machine it is very likely to kill us all." My interpretation is mainly biased on them saying "if we build a really intelligent machine, it is very likely to kill us all."
Yud literally wrote a book titled "If Anyone Builds It, Everyone Dies."[1] There's not much room for ambiguity here...
[0] Yud is on the record saying at least 95% https://pauseai.info/pdoom He also said anyone with a higher P(doom) than him is crazy so I think that says a lot...
Yes, I agree they are saying it is likely going to kill us all. My interpretation is consistent with that, and so is yours. The difference is in why/how it will kill us; you sound to me like you think the rationalist position is that from intelligence follows malice, and therefore it will kill us. I think that's a wrong interpretation of their views.
reply