A Disproof of Large Language Model Consciousness: The Necessity of Continual Learning for Consciousness

It essentially argues that there can be no non-trivial falsifiable theory of consciousness according to which GPT-4 style LLMs are conscious. In this post, I will argue that the argument rests on a computational-complexity-avoiding razzle dazzle.

The paper is quite tricky to read and it took several rounds of arguing with Claude before I realized that most of my initial objections to the paper were already covered under one or the other horn of Hoel’s dilemma. So I’m going to start with a Claude-written summary of the paper. This may be helpful for other readers of the paper.

Can a theory of consciousness be so strongly constrained by requirements for testability that its overall nature can be deduced?

Testing a theory of consciousness is based on comparing two functions that map from observed data to experiences: predictions and inferences.

Experiences (the space E)

This is the target — the set of all possible conscious states a system might be in. Hoel treats it abstractly as a space, E. A particular experience is just one element of that space (seeing red, feeling pain, etc.).

Predictions (the function pred)

A prediction is what a theory says about a system’s experience, based on data about the system’s internal workings. Formally it’s a mapping pred: O → E, where O is data about the system’s mechanics (neuroimaging, the integrated information of the circuitry, the number of layers, whatever the theory cares about). Crucially, what counts as the relevant data is determined by the theory itself. IIT predicts from integrated information; a higher-order-thought theory predicts from the structure of representations; and so on. Theories are individuated by how they make predictions — that’s what makes one theory different from another.

Inferences (the function inf)

An inference is the empirical reason to think a system is having some experience, arrived at independently of any particular theory. Formally inf: O → E as well, but it’s grounded in behavior or report — a button press indicating what someone saw, a verbal claim of pain, or, for an LLM, its text output. Hoel’s key point is that inferences are supposed to be theory-neutral: they’re the “outward” evidence (report, behavior, I/O) that any reasonable observer would take as a sign of consciousness, regardless of what theory they hold. The clearest inference, he notes, would be a system explicitly claiming to be conscious.

Why the distinction does all the work

A theory is tested by comparing pred and inf. If the experience the theory predicts (from internal data) matches the experience inferred (from behavior/report), the theory survives; if they consistently mismatch, it’s falsified. Hoel’s two “horns” come directly from how these two functions relate:

• If pred and inf are fully independent, you can swap in a different system that keeps the behavior/output identical (so inf is unchanged) but changes the internals (so pred changes) — the unfolded network, the lookup table. Now they mismatch everywhere, and the theory is falsified a priori. This is the Substitution Argument.

• If pred and inf are strictly dependent — both drawing on the same source, namely the input/output behavior — then they can never disagree, so the theory can never be falsified. It’s trivial. (Behaviorism is his example: “acts conscious, therefore is conscious” can’t be tested because the prediction is just a restatement of the inference.)

Let’s use multiplication as a concrete example.

Take f to be the times table for single digits: input a pair (a, b) with each in 0–9, output their product. That’s a fixed, finite function — 100 input-output pairs. Now build two devices that compute it:

System A is a Read-Only Memory lookup. Burn a 100-cell memory: cell (7,8) holds 56, and so on. To “multiply,” decode the two-digit address, read the cell. There is no multiplying happening — the answer was precomputed and stored. Feedforward, no intermediate computation, no internal variable that holds a running partial product.

System B is a shift-and-add multiplier — an actual little arithmetic circuit (the kind inside a cheap calculator’s ALU). To compute 7×8 it runs a process: it holds intermediate state, adds and shifts across several steps, propagates carries, updates internal registers, and arrives at 56. There is genuine internal information flow — signals move between parts, partial results get combined.

Both are real chips you can buy or build today. R — the internal structure and dynamics — is wildly different: A is a static store with a one-shot read; B is a sequential process with internal state evolving over steps. And f is identical: every input gives the same output. Type 7×8 into either and you get 56. The inference function, which reads only I/O, cannot tell them apart — inf(A) = inf(B), exactly.

Now run the dilemma, both horns, on this one pair.
Horn one — falsification. Suppose a theory grounds consciousness in something about the internal process: “richness of internal computation,” “amount of information flowing between parts,” “presence of state being updated.” Pick any of those. The theory now predicts System B has some nonzero dollop of it and System A has essentially none — A is just a read. So pred(A) ≠ pred(B). But inf(A) = inf(B), because both just output products. One of A and/or B will have a mismatch, and so the theory is falsified. You didn’t even have to run an experiment — you knew the outputs would match by construction, because you built B’s behavior to equal A’s.
Horn two — triviality. Suppose instead the theory grounds consciousness in the I/O itself: “a system is conscious to the degree it computes products correctly” (or, more honestly, “to the degree it behaves as a competent arithmetic agent”). Now pred(A) = pred(B), because they compute the same products. The mismatch is gone — but only because the prediction has stopped depending on anything inside the box. It’s now just a restatement of what the inference already read off the outputs. Same source, can’t diverge, no scientific information: trivial.

What Hoel tries to do is run the same argument using modern LLMs as system B and a lookup table for the mathematical function implemented by the LLM forward pass as system A.

He concludes that since no non-trivial theory would call a lookup table conscious , no non-trivial theory could call an LLM with the same input-output behavior as the lookup table conscious.

More precisely,

The disproof of LLM consciousness then turns on the claim that an LLM sits so close in “substitution space” to a lookup table that the only properties distinguishing them are ones that, if a theory leaned on them, would either produce mismatches under substitution (falsified) or collapse back onto pure I/O (trivial).

I think the flaw in this argument is that an abstract lookup table does not have input/output behavior. A lookup table together with an implementation on physical hardware has input-output behavior.

This might seem like a trivial distinction in the case of one-digit multiplication. The little circuit B (~1500 transistors) and the 100-cell memory A (~3000 transistors) are both things that comfortably fit in a handheld calculator in your palm.

The times-table function f that maps from an input space of size 10^2 to an output space of 82 (0 to 81) is so small that we hardly make a distinction between the abstract mathematical function and the 100-cell explicitly physically realized version. Hoel’s argument rests on this intuition.

Hoel argues (bolding mine):

Non-conscious universal substitutions for LLMs are well-defined and constructible. In some cases, universal substitutions may be merely conceivable (e.g., consider an infinite lookup table implementing f, wherein f is somehow given). However, in practice, this apparent conceivability may turn out to be ill-defined. E.g., can a human brain be substituted 15 with a functionally-equivalent single-hidden-layer FNN, even one impractically astronomically wide? Is a human’s f clearly definable?

Asking these questions explicitly is not necessarily agreeing that universal substitutions are ill-defined in humans—indeed, the entire rest of the paper will assume these questions are answerable. However, it is undeniable there is significant subtlety and complexity to them. It may be that actual definable and constructible universal substitutions for humans (especially those that count as non-conscious according to Definition 4.2) could be somehow ruled out by details we don’t know or are hard to foresee (e.g., quantum processes in the brain [59], or some computational irreducibility in the process [60], or the no-cloning theorem [61] or some other factor that crops up in an unexpected way).

However, for baseline LLMs, it appears from the literature that universal substitutions are actually constructible via chaining together known methods. Specifically, it is known that transformers can be arbitrarily approximated as RNNs [62, 63]. Then, it is known that (static) RNNs can be “unfolded” into a single-hidden-layer FNN [28, 32], which can in turn be replaced via enumeration via a lookup table.

So the chain discussed in the Proximity Argument (e.g., Theorem 4.6) is literally constructible via known methods, and could potentially even be given as defined transformations. Perhaps for a small-enough language model, some version of it could be transformed this way with current technology. That is, we seem to have a general understanding of how to implement non-conscious (Definition 4.2) universal substitutions for LLMs. This should call into question their consciousness significantly.

Actually the answer is no. We cannot implement non-conscious universal substitutions for LLMs. I did the calculation.

Consider the fairly small large language model Pythia 12b (I chose this model because it’s open source, open weights, and fairly well understood). It is a standard decoder-only transformer with a vocab size of 50688 and a max context window of 2048 tokens. So the mathematical function implemented by a single forward pass of Pythia is

It takes about 40GB of memory to store the LLM version of f_Pythia. Nice and easy. Laptop sized.

The lookup table version, in which we store the output for every possible input, is 1.8 x 10^9641 bytes.

That’s roughly ten thousand in the exponent, to be clear. Another way to write is to say it’s

If we covered every single planet in the entire galaxy with datacenters and nothing but the fanciest datacenters….that would still be about 10^9599 times short. We would have made a microscopic dent in the storage problem, let alone the access and retrieval problem.

Can you build this thing and get some empirical inferences out of it? Hell no. Not if we colonized the Milky Way and devoted all our efforts to testing consciousness papers from Galactic Year 002026.

So my objection is as follows:

The Substitution Argument needs to hold the inference fixed while it swaps the LLM for its lookup-table twin: inf(table) must equal inf(LLM).

Hoel says: “Inferences are, essentially, empirical reasons to believe a system might be conscious irrespective of any particular theory or class of theories.”

An empirical function has to be evaluated on something that actually produces observations. A mathematical abstraction on its own doesn’t suffice.

So to have an inf(table) to hold constant, you’d have to build the table and watch it behave and return outputs to your inputs. For even an LLM as small as Pythia-12b, you can’t. There is no inf(table), so the comparison is undefined.

This non-buildability is half of the escape hatch (from non-consciousness) Hoel reserves for humans, noting that universal substitutions for humans “may turn out to be ill-defined.” The LLM lookup table is mathematically well-defined, but not empirically buildable - and empirical observations are what you need for the mismatch to occur.

Hoel would probably argue that his goal is “a priori falsification”, and the inf(table) is by construction known to be the same as inf(LLM), even if we can’t build it. But if you’re not going to actually run the (thought) experiment, then we might as well talk about the equally unbuildable lookup-tables-for-humans we might be able to conceive of and define in the future.

As for the present, if you actually get to see conscious-looking input-output behavior in real life and make empirical inferences, you can infer that there is some structure behind it, even if you don’t know what it is. More specifically, if you see input-output behavior of the kind you can get from a modern LLM system, it has to be coming from something with some nontrivial internal structure - it can’t be coming from the degenerate “lookup table” version, because that guy doesn’t fit in our galaxy.

Our intuitions that LLMs are not conscious comes from the fact that we understand them fairly well at a mathematical level. And so the mystery of where their “divine spark” comes from is somewhat dissolved. For humans we don’t really understand what humans are doing very well at all and so there’s lots of mystery there. To be sure, humans are also much, much more complicated and are doing way more intricate and integrated things than LLMs are. But with time and scientific progress we will dissolve the mystery eventually, and then we will have to reckon with disproofs of human consciousness. We may as well undisprove them now.

https://julianrdcosta.com/the-simplest-explanation-of-expectation-maximisation/

Say the wife asks the husband "I'm so stressed with all the stuff I have to do and I'm so tired, can you help" and the husband says "sure honey, just tell me what to do and I'll do it".

This ends up not working, either because the wife gets pissed at this point, or she says something like "okay can you handle making the kid's tiffins for school". The husband does it, but forgets sometimes or fails to meet the wife's standards, and the wife feels she has to still made sure the kids actually get tiffins, so she feels not-helped and even more stressed, and the husband also feels aggrieved and unappreciated.

I was wondering why this dynamic occurs. And I realised the core reason is that it's not common knowledge that project management is the hardest part . This is an idea I got from Visa Veerasamy, in a thread where he points out that many video games are simulations of work - tasks that people normally only do if they’re paid to do it. But they do it for free in video games and enjoy it - because the game tells you what you need to do and exactly when. There’s no friction of waiting on stakeholders, negotiating unclear requirements, or other people getting mad at you when things go wrong or not as planned.

In the case of household tasks we started with, the wife is burdened by responsibility - she wants to delegate the whole task of ensuring it’s done, not just the physical execution part. But she complains about the execution - and so the husband thinks1 "aha i can execute, no problem, easy peasy".

The complication occurs when the two have different standards for the task. The worst-case scenario is when the wife cares more about the results of the task or will face greater social punishment for it not being done well, or just is more particular, and resorts to micromanaging the husband's doing of the task. Which means that she's even more stressed, while the husband feels annoyed at being micromanaged, and also annoyed at her annoyance because what's the problem? He's helping, he's doing it, why are things worse than before, which is the start of a doom spiral.

Solutions are, of course, left to the reader.

Inspired by:

Eurydice Lives

Collaborators and Privates

The other day in a Note, I was speculating on answers to a question from John Williams PhD: why breadwinning women might divorce their spouses. I offered that I think when women work harder, they expect certain behavior from men based on their own labor division strategies. In cases where these behaviors aren’t forthcoming from their husbands, women may leave…

Read more

a year ago · 71 likes · 18 comments · Eurydice

Subscribe now