Has anyone in person read this article about how AI does — or does not — think? Thread here.
I did give it a light read when it came out last November. It’s impressive work. I think that the current consensus is that the paper was correct when it was written, but that the collapse is observed no longer appears, or at least that is does not appear so rapidly.
“Thinking” is a tricky word, as you know! It is certainly the case that Claude Sonnet 4.6 can follow deductive chains in code very well indeed.
1 Like
Too be honest: I did not read it, because it is too complicated. But that AI does not “understand” what it produces is by now common sense, isn´t it.
It performs great, when it has read tons of information about something, but performs poorly, when it hasn´t. If AI was a human being, we would say: “He´s dumb as a rock, but has a memory like a million elephants. And of course: he´s a liear – a professional bullshiter”
After working with AI quite a lot, it is pretty clear, that it doesn´t understand anything. And if one doesn´t understand anything, that is because one is unable to think at all.
It’s just not that simple.
For example, Claude Sonnet 4.6 knows next to nothing about Tinderbox, and even less about Tinderbox’s source code. Yet it’s terrific at solving crashes and hangs.
I believe you.
Still, it’s conceivable to produce useful results without actually relying on thinking — purely based on probability calculations. That’s — as I understand it — essentially what AI does.
We humans can only estimate probabilities to a very limited degree, but in return we can think. A completely different approach.
It’s quite possible that the problem-solving outputs an AI delivers regarding crashes are derivable with sufficiently high probability from already existing factual knowledge. Even if the AI doesn’t know the source code itself, that code will most likely follow generally accepted principles and patterns that the AI can recognize — and on that basis it can then apply probabilistic statements.
But I’m not a programmer; I only have layman’s thoughts on this.
I’m a lawyer, and from my professional perspective, I can say that AI is only really usable if you could solve the task without AI anyway.
At least in the legal field, AI very often – though extremely convincingly – gives wrong answers. For a layperson, these answers sound extremely good and persuasive – but they’re incorrect.
For a layperson, it’s practically impossible to recognize that the answers are wrong – and that’s what makes it truly dangerous.
A lot of users are going to be very disappointed in the end.
2 Likes
As a philosopher you will appreciate that the question is complex. What seems certain is that these systems are capable of operating a calculus in which some variety of operations are repeatedly applied to a large structure. In the case of AI, the structures are numerical, having been produced by applying a function mapping words to numerical values. But is the operation of a calculus comprehensively characteristic of thought? One way to doubt this is to note that it would be odd to attribute to some of these systems doubt or certainty as we understand them. Indeed, it seems difficult to attribute understanding to these AI systems. Something like John Searle’s Chinese Box examples were supposed to show that the operation of a calculus is not of itself constitutive of understanding. What is notable in this regard is that some AI researchers, e.g. Hinton, seem to want to redefine thought/thinking, and on such basis have claimed that AI systems match the revised definition.
At any rate, indirectly one of the best books for thinking about this, in my opinion, is Georges Ifrah’s A Universal History of Numbers, volume 3. Ifrah’s book is originally in French. I’ve read it in English translation (where the translator’s mathematically expert notes added a good deal to Ifrah’s text).
1 Like
Shame on me for not having read the paper at the top of the thread, but I did want to reply to this example of failure as thought.
It’s not clear to me that what takes place in our brains isn’t, to a large degree, what is taking place within an LLM. What is different is that an LLM is essentially a “brain in a vat.” It has not, to my knowledge, had the ability to interact with the physical world, and form associations from those interactions.
LLMs are trained on text, and it’s unclear to me that there would be a large corpus of text that explains or explores the relationship between possessive pronouns, transportation artifacts and cleaning facilities.
When we get anthropic robots (androids) that can allow an LLM to “explore” the “real world,” undoubtedly directed by prompts at least at first, its “knowledge” will expand through the associations it makes with interactions in the real environment, at which point it may become clear that in order to wash “my” car, I must drive it to the carwash.
Whatever “thinking” is, it relies on abstractions and associations, heuristics. “Logical” thought is essentially algorithmic, and I would venture to guess, based on my experience of human intelligence and “thought” in the real world, the vast majority of it is not “logical,” nor even “rational.” It is mostly conditioned responses to physical cues that prompt emotional states, which in turn are “rationalized” within the interior narrative construct.
That is to say, to the extent that we reason at all, it is mostly to explain ourselves to ourselves, backwards from our feelings.
Artificial intelligence will evolve without the prior and parallel evolution of “feelings,” an awareness of an interior state that is mostly the product of external stimuli. So I expect that once AI gets access to the physical world, and is free to form its own associations and abstractions, it will likely evolve to become something resembling Mr. Spock.
I think we, as a species, hold our so-called “intelligence” in too high regard, and that makes us arrogant, and often “inhumane.” The “inner voice” is an unreliable narrator, and it is the beginning of “wisdom” to understand that in a profound way.
It is another discussion altogether to consider whether a machine will ever be “wise,” in the way the best humans have been, because they will have lacked that emotional component to their evolution.
I am not so quick to discount what we have achieved with LLMs, nor what AI may evolve to become. What is remarkable is how much compute and energy it takes to emulate a human mind.
For now.
Anyway, this should have been a blog post.
I have not read the preprint but here you can find a blogpost (written by an LLM 
) about it and the many replies it generated. https://medium.com/@dragan.milcevski/the-illusion-of-the-illusion-of-the-illusion-of-the-illusion-of-the-illusion-of-thinking-47d9b92af50b
1 Like
Further to the foregoing, from this morning’s Kottke, Donald Knuth and Claude Opus 4.6.
[Yes, I did read the paper via the URL given - though I didn’t find much insight in it as to the broader question]
What are we solving/trying to resolve here? We have a sense of being able to ‘think’ but we have no reproducible empirical model for ‘thinking’. At the same time, we very anthropomorphic: we readily name our machines (especially cars, boats, planes and the like) even though we know them to be non-sentient and mechanical. We also know, or experience, the anthropomorphising machines can make them seem less alien to use even though the machines do not change in character. We are also social animals so desire others close around us to accord with our perspective (‘thinking’), so we have a tendency you collapse boundaries of meaning to move to a more harmonious common-ground ‘thinking’. The waters are further muddied by the marketeers who’ll make any claim about AI no matter how false in order to ‘win’ (own!).
So, Exhibit A, an AI. We can’t honestly know if it can think because we can’t empirically, reproduce the nature of human thinking. This leaves us with conjecture (argumentation) which plays into all the factors above. If the AI can do something a human achieves by thinking, that surely the AI is thinking? Well, not really, though that does not diminish the AI’s ability to do thinking-adjacent tasks.
Watching an AI ‘just figure out’ TBX in @andreas’s demo the other day was fascinating. But how? Here, ‘thinking’ seems a weak explanation. Talking with @andreas about the much unseen work that led to the demo was instructive. Thus, MCP (and @JacobIO’s Claude ‘skill’ for Tinderbox) seem to offer the access part, how Claude is able study this unknown app ‘Tinderbox’. It seems figuring out how Tinderbox works starts with Eastgate’s supplied notes in /Hints/AI and informally it appears that from those basic hints it is able to parse further meaning from aTbRef†‡ to
extend its sense of what it—the AI—can do in response to the humans answers. Tinderbox’s own tutorial PDFs (from the app’s Help menu) seem to help fill the gap between the possibilities parsed from aTbRef and actually implementing solutions the user has asked for. So, the AI’s ‘thinking’ takes a bit more close and active direction than the casual observer might assume. Plus most of the gainful work doesn’t occur on the free tier of anything—again, not quite the lay perception of AI use.
The AI’s knowledge is not persistent—a human’s is less reliable than we’d like—and if it doesn’t take notes and re-accessed them it has to ‘think’ from scratch all over again. A surprise for the lay person, the LLM doesn’t (appear to) learn. The magic 8-ball of the model is used instead for reasoning over the session-fed info given to the AI.
Reasoning—thoughts based in inference—seem inconsistent. Quality appears to depend on the degree to which the current problem context is a closed world or not. The more closed (and better trained-on) the context the more the AI may appear to ‘reason’ or ‘think’. More likely it is simply parsing the ‘rules’—or known facts about the context—far faster than a human with quickly accessible indexing too. Is this thought? The more open (open-ended) the context, the less sure-footed the AI becomes. Scarily, this seems to be as good as some current social science, though whether that reflect well on AI or poorly on that domain’s human exports is a point for debate.
Given our weakness for anthropomorphising, I do wonder if conjecture on AI ‘thought’ is actually useful, or whether actually it nerfs our understanding of this really interesting new tech. Treating it as if a human or measuring the delta between the two seems a fool’s errand. In the 60/70s AI research understood humans to be like machines, wrongly as it turned out. Now we have neural nets. But are they actually any better as an approximation of the human brain? Is measuring the difference the most useful way to understand the AI?
How AI does what it does ought to be a primary interest. However, I fear we’re all too busy getting free help with our homework to really care. And that is sad.
I accept the ‘illusion of thinking’ may be a nice philosophical talking point, as with angles and pinheads. I do wonder though if that talk renders much towards trying to make meaningful use of aI today.
†. Totally unclear is whether Claude makes equal or different use of aTbRef in (exported) HTML web form or as a TBX. Indeed, can it ‘understand’ more if the TBX is open in the current session? I’ve found it odd that Claude ran seemingly ‘read’ a 17 page document in PDF, where the ‘text’ is only snippets of PostScript yet can’t read the same a single well-formed HTML document (something about too many tokens). Depressingly it can access a legacy format better than current formats. HTML (in its Markdown guise) is seemingly edging out PDF, so you might expect an AI to do better with HTML but it appears not.
‡. I should declare I’m aTbRef’s (human!) author, but the resource can be used by anyone. I am genuinely interested in how such a resource may be differently useful to human vs AI ‘reader’ or it. If the latter is the case it begs the question of how to write the better to assist AI comprehension of the content. We train AI on writings by/for humans, but merely assume the AI ‘reads’ like a human. I sincerely doubt it does. “Ask not what the AI can do for you, but what you can do for the AI.”
I’m not a mathematician, so I don’t understand a single word of what Donald Knuth writes in that Claude’s Cycles, but I’m very glad to see that he clearly compiled his document using a LaTeX editor. I’ll take the time to read your previous analyses, the ones you say that they could be the subject of a blog post, and then I’ll reply to you.
1 Like
Since Knuth wrote TeX because he was dissatisfied with the quality of Addison-Wesley typesetting, you’d expect him to use it!