They don't have any concept of falsehood either, so this is very different from a human making things up with the knowledge that they may be wrong.

I think the first part of that statement requires more evidence or argumentation, especially since models have shown the ability to practice deception. (you are right that they don't _always_ know what they know)

But sometimes when humans make things up they also don't have the knowledge they may be wrong. It's like the reference to "known unknowns" and "unknown unknowns". Or Dunning-Kruger personified. Basically you have three categories:

(1) Liars know something is false and have an intent to deceive (LLMs don't do this) (2) Bullshitters may not know/care whether something is false, but they are aware they don't know (3) Bullshitters may not know something is false, because they don't know all the things they don't know

Do LLMs fit better in (2) or (3)? Or both?

It's super interesting.

There are two levels...

The pZombie type level where we look at the LLM as if it were a black box and simply account for its behavior. At this level LLM's claim to have knowledge and also claim knowledge of their limited knowledge "I can't actually taste". So approached from this direction we are in (2) they have awareness that there are some things that they don't know, but this awareness doesn't prevent them from pretending to this knowledge.

If we consider it from the perspective of knowing what's happening inside LLM's then I think the picture is different. The LLM is doing next word prediction with constant compute time per token - the algorithm is quite clear. We know this is true because it runs on llama.cpp or mlx on our macbooks as well on the farms of B200's that we fear will destroy the atmosphere. So LLM's don't have any actual operational knowledge of the logic of their utterances (dunning kruger, dunning kruger...) What I mean is that the LLM can't/isn't analysing what it says, it's just responding to stimulus. Humans do do this as well - it's easy to just chatter away to other people like a canary, but humans also can analysis what they are saying and strategically manipulate the messages that they create. So I would say that LLMs cannot be concerned about what they do or don't know - the concern rests with us when we challenge them (or not) by asking "how can you know that chocolate tastes better than strawberry - you have never tasted either".

But you can combine them with something producing truth such as a theorem prover.