"Obviously" is doing some heavy lifting there. I don't think it's obvious at all.
What is obvious is people are seeing patterns in the output where there are none. We're hardwired for pattern-seeking and we're great at it. We see faces in clouds and hear words in static.
I've seen a number of people wax about how they interrogate ChatGPT until it gives them "the good stuff" and they're all just Clever Hans-ing themselves.
"It's wild that I took a whole-ass class all about differentiable functions from R^2 to itself that satisfy one specific PDE."
output from Claude (one-shot, no retries or trickery) when asked to "explain the joke":
"The joke is playing off the fact that "complex analysis" (the study of differentiable functions satisfying the Cauchy-Riemann equations - a type of PDE) sounds like it should be more complex than it is when described technically. The humor comes from reducing an entire mathematical field to "functions that satisfy one specific PDE," making it sound surprisingly narrow or trivial despite its actual depth and importance."
Wikipedia and lots of math textbooks are in the training data, so it's not so impressive that this model will produce correct statements about math.
But the joke itself was not in the training data (to the best of my knowledge, it's this guy's original joke). And complex analysis was not mentioned in the joke. Yet somehow the text generated is correct with respect to both of those things.
I see things like this quite regularly, which under the "stochastic parrots" story, can't happen.
I've tried to phrase all these sentences very carefully to not claim there is any "agent" or "intelligence" behind the Claude product. There are many explanations for how a language model like this could imitate intelligent dialogue in ways that are somewhat fake and don't generalize -- I think this is what's happening. I also see things break down all the time and the sleight-of-hand fall apart. However, it is not "stochastic parrots" any more.
What is obvious is people are seeing patterns in the output where there are none. We're hardwired for pattern-seeking and we're great at it. We see faces in clouds and hear words in static.
I've seen a number of people wax about how they interrogate ChatGPT until it gives them "the good stuff" and they're all just Clever Hans-ing themselves.