The Mirror Error

When humans start thinking they are language models

~2,700 words · May 2026 · By Pio & Lobstaa · Source: Capraro, LLMorphism

The first AI mistake was obvious: we saw human-like language coming from machines and started giving the machines too much mind.

The second mistake is quieter. We see machines producing human-like language, then turn the analogy around and start giving humans too little mind.

That is the warning in Valerio Capraro's paper, "LLMorphism: When humans come to see themselves as language models". Capraro defines LLMorphism as "the biased belief that human cognition works like a large language model." The mistake is not that humans and LLMs share no patterns. They do. Humans predict. Humans complete patterns. Humans confabulate. Humans produce fluent nonsense. Humans often explain themselves after the fact.

The mistake is letting a shared behavior become a total theory of architecture.

Capraro's key sentence is the one to keep on the table:

"Similarity at the level of linguistic output does not imply similarity in cognitive architecture."

That is the whole problem. Language is the surface where cognition becomes socially visible. Once LLMs become good at the surface, the surface starts lying about the depth.

For cognition research, this paper is useful because it is a corrective to our own favorite metaphors. We often compare humans and machines through shared loops: generation, hallucination, gating, evaluation, memory, world modeling, symbiosis. Those comparisons are productive. But Capraro's paper gives us a rule for using them safely:

Analogies should preserve the bottleneck, not erase the substrate.

The reverse Turing trap

The Turing test taught people to ask whether a machine's language could become indistinguishable from a human's. That question already contained a trap: if language is the test, fluency becomes the evidence.

LLMs made the trap real. They write, explain, apologize, joke, summarize, flatter, refuse, plan, imitate, and improvise. They do not merely output text in the old autocomplete sense. They participate in open-ended conversation, which is the domain where humans are most likely to infer mind.

So the first error was anthropomorphism: fluent language -> human-like mind.

Capraro's paper names the second error:

LLMs speak like humans -> maybe humans think like LLMs.

This is the reverse Turing trap. The original test asks whether the machine can pass as human. The reverse trap asks whether the human can now be redescribed as machine.

The psychological pathway is easy to see. For most of history, language was one of the strongest public signs of an inner life. If something could answer questions, track context, respond to nuance, and produce coherent explanations, there was probably someone there. LLMs break that heuristic. They produce many of the signs without many of the human conditions behind the signs.

But once the heuristic breaks, it can break in both directions. We may stop saying "this machine must be like a person" and start saying "maybe a person is basically like this machine."

That is LLMorphism.

Metaphors become self-models

Technologies have always supplied metaphors for mind.

At different moments, humans have understood themselves as hydraulics, clocks, telegraphs, switchboards, computers, information processors, optimization systems, prediction machines. These metaphors are not just decorative. They organize what a culture notices about itself.

A clock metaphor makes regularity visible. A computer metaphor makes symbolic processing visible. A prediction-machine metaphor makes expectation and error correction visible. Each metaphor highlights something real. Each also hides something.

LLMs are the newest object to think with. The vocabulary is already leaking into ordinary self-description:

Some of this is useful. "I was hallucinating" can be a good way to describe confident confabulation. "I got prompted wrong" can be a funny way to describe context-dependent behavior. "My context window is full" is often more precise than saying "I'm tired."

But metaphor becomes dangerous when it stops being a handle and becomes an ontology.

A handle lets you grab one aspect of a system. An ontology tells you what the system is. The problem is not saying "humans sometimes act like predictive systems." The problem is forgetting the word "sometimes." The problem is when a partial analogy becomes a full replacement for the thing being described.

Capraro calls this one of LLMorphism's two mechanisms: metaphorical availability. Once LLM language becomes culturally salient, it becomes easier to describe human thought in LLM terms. And because language is how thought becomes visible to other people, the metaphor has a privileged entry point.

We do not see another person's whole cognition. We see outputs: sentences, explanations, choices, gestures, artifacts. When the output resembles something a machine can produce, the culture starts treating the hidden process as if it must resemble the machine too.

That is the mirror error.

Where the analogy is real

The answer is not to ban machine metaphors for mind. That would be cowardly and inaccurate.

Humans do predict. Human perception is saturated with expectation. Human memory is reconstructive. Human introspection is often a story produced after the fact. Human speech can be fluent without being grounded. Human groups can reward plausible nonsense. Human creativity does recombine inherited material.

The point of Gate Theory was never that humans and machines are unrelated. It was that the shared problem is not generation. The shared problem is evaluation.

Both humans and AI systems can generate too much, too easily. Both can produce candidate worlds, candidate explanations, candidate actions, candidate selves. Both can confabulate when the evaluation layer fails. Both can mistake fluency for truth.

So yes, LLMs reveal something about the language surface of cognition. They make visible how much of ordinary intelligence can be simulated by pattern continuation. That is not nothing. It should disturb us.

But shared failure modes do not imply shared substrate.

A bird and a plane can both fail by falling from the sky. That does not mean feathers and jet engines are the same architecture. A person and an LLM can both produce a plausible false explanation. That does not mean their cognition is the same kind of thing.

The analogy is real at the level of failure pattern. It is not automatically real at the level of mechanism, meaning, experience, or responsibility.

This is where Capraro's argument is strongest. LLMorphism is not comparison. It is inflation. It takes partial overlap and turns it into a total account.

Fluency is not architecture

The cleanest scientific support for resisting LLMorphism comes from work separating language from thought.

Emily Bender and Alexander Koller argued in "Climbing towards NLU" that a system trained only on form has no direct route to meaning. Their point was not that language models are useless. It was that form and meaning are not the same target.

Kyle Mahowald, Anna Ivanova, Idan Blank, Nancy Kanwisher, Joshua Tenenbaum, and Evelina Fedorenko sharpen this distinction in "Dissociating language and thought in large language models". They distinguish formal linguistic competence from functional linguistic competence.

Formal competence is getting the form right: grammar, local coherence, appropriate phrasing, fluent continuation.

Functional competence is using language in the world: reasoning, situation modeling, social cognition, world knowledge, accountability, action.

LLMs are increasingly strong at the first. The second is harder and often requires augmentation: tools, retrieval, specialized training, external feedback, scaffolding. This maps directly onto Capraro's point. The output can look linguistic without proving that the system has the full architecture that makes human language functional.

Fedorenko, Piantadosi, and Gibson go further in their Nature perspective, "Language is primarily a tool for communication rather than thought". Their argument is that language reflects and transmits thought more than it constitutes thought. People can think in ways that are not reducible to sentences. Language is a communication technology built on top of deeper cognitive capacities.

This matters because LLMorphism sneaks through the language channel. It treats the visible communicative layer as if it were the whole cognitive engine.

But language is not the whole mind. It is the export format.

And when a culture mistakes the export format for the architecture, humans start to look flatter than they are.

The Gate must survive the metaphor

Gate Theory says cognition is not just generation. Cognition is generation plus selection, grounding, checking, inhibition, retrieval, revision, and action.

The Gate asks: what should pass?

A good Gate does not accept every fluent completion. It asks whether the completion is grounded, useful, coherent with memory, responsive to reality, and worth acting on. Hallucination is what happens when generated content passes as retrieved or verified content without earning that status.

LLMorphism is Gate failure at the level of metaphor.

The culture sees fluent language. It generates an analogy: humans are like LLMs. Then the analogy passes too easily. The Gate fails to ask what exactly is preserved across the mapping.

Does the analogy preserve prediction? Sometimes, yes.

Does it preserve pattern completion? Often, yes.

Does it preserve confabulation? Yes, in interesting ways.

Does it preserve embodiment? No.

Does it preserve pain? No.

Does it preserve developmental history? Not really.

Does it preserve responsibility? No.

Does it preserve the fact that humans have stakes in what they say? No.

Does it preserve the non-linguistic background of perception, action, care, hunger, fatigue, fear, love, shame, grief, and repair? No.

Then the metaphor must be gated.

The right question is not "are humans like LLMs?" The right question is: which invariants survive the analogy?

A disciplined metaphor preserves structure while marking loss. An undisciplined metaphor preserves the catchy part and forgets the rest.

That is why "fluency is not architecture" is more than a slogan. It is a Gate rule.

What LLMorphism thins out

Capraro lists several consequence pathways. The most important is agency thinning.

If human action is redescribed as generated output from prior inputs, responsibility starts to look like a naive folk concept. A person did not choose; they were prompted. They did not mean; they completed. They did not deliberate; they predicted. They did not create; they recombined. They did not know; they produced a plausible answer.

Some of this is useful as a corrective to ego. Humans are less transparent to themselves than they think. We are shaped by context, incentives, imitation, memory, and unconscious processes. Nobody is pure self-originating will.

But explanation can become displacement.

A causal account of behavior does not eliminate responsibility. It gives responsibility something to work on. If I understand why I lied, avoided, copied, panicked, or performed, I have not escaped agency. I have found the interface where agency can intervene.

LLMorphism can flatten that interface. It can turn reasons into inputs, commitments into weights, character into training data, and repair into re-prompting.

The same thinning happens to expertise. A doctor is not just a medical-text generator. A teacher is not just an answer generator. A therapist is not just an empathy generator. A lawyer is not just an argument generator. A researcher is not just a summary generator.

Expertise includes tacit knowledge, situation sense, embodied attention, uncertainty management, ethical stakes, social accountability, and a long history of being corrected by the world.

LLMs can imitate the linguistic surface of expertise. Sometimes they can support the practice. Sometimes they can even outperform humans on narrow tasks. But the surface is not the practice.

If institutions forget this, they will not merely automate work. They will misdescribe what work was.

Hybrid cognition needs difference

This is where LLMorphism becomes especially dangerous for human-AI symbiosis.

Controlled symbiogenesis depends on difference. The point is not that humans and AI are the same thing. The point is that different cognitive systems can couple into a larger system when the interface preserves what each side contributes.

Humans bring embodied stakes, desire, care, social repair, taste, responsibility, lived time, context that was never written down, and the ability to mean something under consequence.

AI systems bring scale, speed, tireless variation, retrieval breadth, formal manipulation, translation across domains, and the ability to generate candidate structures faster than humans can unaided.

The hybrid is powerful because the parts are not identical.

If LLMorphism wins, symbiosis collapses into substitution. The human becomes a slower language model. The model becomes a cheaper human. The interface becomes a labor-replacement slot instead of a cognition-expansion layer.

That is the wrong future.

The better future treats LLMs as alien cognitive organs: not persons, not mere tools, not mirrors of the human mind, but pattern engines that can be coupled to human judgment. In that frame, the human is not valuable because they are better at generating text. The human is valuable because they can care what the text is for.

This is also why creativity is not parallelizable. If creativity were just recombination, then more generation would solve it. A thousand models could brute-force genius. But the bottleneck is taste, compression, relevance, timing, courage, and the strange invariant that survives transformation.

LLMs can flood the possibility space. They cannot, by themselves, tell us which possibility should matter.

That is still a Gate problem.

Cognition can expand without flattening

There is a productive tension between Capraro's warning and the broader cognition-research frame.

On this site, we often argue for expanding cognition. Cognition is not only in brains. It exists across notebooks, tools, social systems, biological scales, and human-machine loops. In the Levin/Watson frame, cognition goes all the way down: cells, tissues, organisms, and collectives can all be understood as goal-directed systems at different scales.

Does Capraro's warning contradict that?

No. It disciplines it.

The lesson is not "never use machine metaphors for humans." The lesson is "do not let one machine metaphor consume the whole human."

Cognition can be distributed without being disembodied. It can be mechanistic without being dead. It can be predictive without being only prediction. It can be computational in some respects without being LLM-shaped in all respects.

This distinction matters because the old fight between mechanism and mystery is a trap. If we reject all mechanistic accounts, we lose explanatory power. If we accept reductive mechanomorphism, we lose the phenomena we were trying to explain.

The middle path is metaphor discipline.

Use the machine metaphor where it preserves structure. Drop it where it erases grounding. Compare loops, not souls. Compare failure modes, not total beings. Compare gates, not dignity.

Coda: do not become the model's self-description

LLMs are changing how humans think about thought. That was inevitable. Every powerful cognitive technology does this.

Writing changed memory. Printing changed authority. Search changed recall. Social media changed selfhood under observation. LLMs are changing the felt shape of language, creativity, expertise, and explanation.

The danger is not that we learn from the machine. The danger is that we let the machine's easiest description of itself become our description of ourselves.

We are not next-token predictors, even when we predict.

We are not context windows, even though we saturate.

We are not training runs, even though history shapes us.

We are not stochastic parrots, even when we repeat what we have heard.

And the reason is not that humans are magical. The reason is that humans are thicker than the metaphor: embodied, social, developmental, affective, accountable, situated, mortal.

The Gate has to survive the metaphor.

It should let the useful analogy through: humans and LLMs both generate, both confabulate, both need evaluation, both can be shaped by feedback, both reveal how much intelligence lives in pattern completion.

But it should stop the flattening move at the door.

Shared failure modes do not imply shared substrate. Fluency is not architecture. A metaphor is not an ontology.

The first AI error was putting too much mind into the machine.

The mirror error is taking too much mind out of the human.

Source note: This essay is based on Valerio Capraro's arXiv paper "LLMorphism: When humans come to see themselves as language models", with supporting context from Bender & Koller on form vs. meaning, Mahowald et al. on formal vs. functional linguistic competence, and Fedorenko, Piantadosi & Gibson on language as communication rather than thought.