The Beast In The Web

Reviewing Anil Seth’s Being You: A New Science of Consciousness

I. WHAT

Somewhere, on a far off hill in Arkansas, John stumbled across an infinite-dimensional object. Not a 4d hypersphere, but one with uncountably infinite sides.

Of course, he didn't know it was an infinite dimensional object—in 3d, it just looked a bit odd—and neither did the scientists that came in hordes to examine it. They measured it from every direction, in every way, with every wave of light and sound and particle they could fire at it.

It was a brilliant young scientist from Belgium that realized—through an elaborate mathematical exploitation of the newly discovered relativity—that the object appeared slightly different from each reference frame. Not just inertial reference frame, subjective reference frame.

So people flocked to take photographs, draw sketches, write poems. They tried to piece them together, but found these fragments resistant like trying to join puzzle pieces made from steam. It was, after all, infinite-dimensioned.

However, over the years, new theories emerged: the hyper-object was connected, supposedly, to the ruminating echoes of the universe's birth, to the donut-hole of galactic spacetime, to string theory, to warp-drives; it was a window into the true moral plane, hypothesized the legendary philosopher Kannot. Eventually, something stuck: it turned out the object encoded some form of causal relation with the fundamental principles of reality, a connection that scientists discovered could be measured directly through a fifth fundamental force linking it to every atom in the universe, and reversed to harness the infinite zero-point energy of the hyperspace.

It only took one declaration of "the biggest hyper-thermometer ever made" to set off an arms race. The first one stretched from Geneva to Milan, the next between San Francisco and LA, the seventh New Zealand to Singapore. 10 years in, mega-corporations were constructing arrays nearing the size of mercury. Everyone wanted to know the hidden secrets of the universe. And the endless energy offered meant everyone was rushing headlong to find out.

It was Alien Thes—an extraterrestrial physicist from the neighbouring galaxy perplexed by the milk way’s suddenly broken silence—that first raised the question everyone had seemingly forgotten:

This object, after everything, did it not still have infinite dimensions? Even with every atom in the entire universe thrown at it, could humanity ever truly capture it?

II. WHO

On an episode of Alex O'Connor's podcast Within Reason, titled somewhat provocatively Why AI will Never be conscious, Alex poses a question: "When we talk about AI being conscious, what do we really mean?"

Situated alongside his questions of where possibly this consciousness could be situated (your phone? my phone? a server? is it like a conjoined twin?), it was attempted primarily as a rhetorical gesture toward highlighting its own absurdity. The episode's guest, Anil Seth, did not immediately discard it as such. He gestured toward the relevant questions, such as "is there an identity? What is the identity over time? What's actually talking to you? It's not the foundation model because that's stable. Is it an instance? Is it a server farm?", but eventually settled on Alex's conclusion of rhetorical absurdity:

I think the extent to which it's difficult to make sense of what we would even mean by conscious AI could be an indication that the thing we're talking about is a little bit senseless, maybe.

This essay is an attempt to address that question. Or, more specifically, it is an attempt to address the question of where Seth's answer came from.

Because Anil Seth's perspective tracks back to a perspective on consciousness first laid out in his 2021 book Being You. It was here that he asserted himself strongly in the camp of biological naturalism, concluding with an “only an intuition” that AI likely cannot be conscious.

Across his paper Conscious Artificial Intelligence and Biological Naturalism and award winning essay The Mythology of Conscious AI his take transitions from potentially not to very likely not. While his explicit positions often remain hedged, even in the Alex O'Connor podcast he slips in a "maybe", the impact of his arguments are massively influential.

His book Being You achieved: Sunday Times Top 10 Bestseller; book of the year for The Economist, New Statesman, Bloomberg, and Five Books; Science Book of the Year for The Guardian and Financial Times; and Wall Street Journal said "If you only read one book about consciousness, it must be his." His 2017 TED talk received over 15 million views, he has received the Royal Society's Michael Faraday Prize for his writings and engagement with science, and has been cited by the CEO of Microsoft AI to support the position that AI consciousness is not worth investigating.

So what does Seth actually believe, and how well does it hold up?

III. HOW

The core thesis of Being You is Seth's Beast Machine Theory:

Our conscious experiences of the world around us, and of ourselves within it, happen with, through, and because of our living bodies. Our animal constitution is not merely compatible with our conscious perceptions of self and world. My proposal is that we cannot understand the nature and origin of these conscious experiences, except in light of our nature as living creatures.

He splits his argument for this into a clean, four part structure: Level, Content, Self, and Other.

The first part, Level, establishes his foundational methodological perspective on consciousness: that we should replace the famous and self-evidentially difficult Hard Problem of consciousness with his proposed Real Problem, by focusing on consciousness science that can "explain, predict, and control the properties of conscious experience". He also explores some of the emerging work being done in attempts to measure a qualitative “level” of conscious experience with methods grounded in theory, rather than fallible behavior or self-reports.

Part II, Content, builds up predictive processing, Bayesian inference, precision weighting, and active inference—likely familiar to ACX readers from Scott's review of Clark's Surfing Uncertainty—before arriving at the chapter's genuine payoff: that the deep structure of perceptual experience itself (objecthood, change, time, causality, the very sense that things are real) is inherently this constructed inference all the way down, rather than any direct contact with raw perception.

Self builds upon this to further claim that our deeply-entrenched sense of self—alongside being truly fractured into bodily selfhood, perspectival self, volitional self, narrative self, social self—is primarily our cognitive predictions turned inward. This culminates in his Beast Machine theory: "All of our perceptions and experiences, whether of the self or of the world, all are inside-out controlled and controlling hallucinations that are rooted in the flesh-and-blood predictive machinery that evolved, develops, and operates from moment to moment always in light of a fundamental biological drive to stay alive. We are conscious beast machines, through and through."

Seth ends Being You by stepping back in Other, by examining the implications of his new theory, first across the diverse range of potentially conscious non-human animals, and then to decide about AI that "there are no conclusive reasons to believe that they actually are conscious".

The Real Problem

Part of Being You is in response to the Hard Problem of Consciousness, originally formulated by philosopher David Chalmers in 1994. In Chalmers’ own words:

It is undeniable that some organisms are subjects of experience. But the question of how it is that these systems are subjects of experience is perplexing. Why is it that when our cognitive systems engage in visual and auditory information-processing, we have visual or auditory experience: the quality of deep blue, the sensation of middle C? How can we explain why there is something it is like to entertain a mental image, or to experience an emotion? It is widely agreed that experience arises from a physical basis, but we have no good explanation of why and how it so arises. Why should physical processing give rise to a rich inner life at all? It seems objectively unreasonable that it should, and yet it does.

Seth believes this framing is misguided, and points to the problem of vitalism that once plagued life to suggest why:

Vitalists thought that the property of being alive could only be explained by appealing to some special sauce: a spark of life, an élan vital. But as we now know, no special sauce is needed[...] The fatal flaw of vitalism was to interpret a failure of imagination as an insight into necessity.

He believes that the science of consciousness will go through a similar shift, and does not claim to solve the hard problem but instead that it will dissolve as we continue to focus on the real methods through which conscious experience functions—what he calls the "Real Problem" of consciousness:

What counts as mysterious now may not always count as mysterious. As we get on with explaining the various properties of consciousness in terms of their underlying mechanisms, perhaps the fundamental mystery of 'how consciousness happens' will fade away, just as the mystery of 'what is life' also faded away.

The Beast Machine

The phrase "Beast Machine" is Seth reclaiming the phrase bêtes-machines, which Descartes originally used to claim that non-human animals lack the res cogitans (mind-stuff) necessary for consciousness and are "merely machines that breathe, digest, perceive and move by means of the arrangement of parts." Seth is inverting the framework: yes, we are organic beast machines. But we are conscious precisely because of that, not despite it.

His argument begins by establishing that the brain does not just receive sensory input—used for predictive error correction—from the outside world (exteroception), but also from inward-facing interoceptive signals of "how good a job the brain is doing of keeping its body alive." He highlights how, just as predictive processing establishes the brain lacks access to the true outside world and so must project a continuously reconstructed model, the same is true of our own body. The brain, stuck in the dark electricity of your skull, can only infer physiological regulation by predictively modelling its own body.

Seth draws upon a few principles from cybernetics to support this. First, the 1970 Good Regulator Theorem: EVERY GOOD REGULATOR OF A SYSTEM MUST BE A MODEL OF THAT SYSTEM. Active inference has two possibilities to reduce predictive error: you can change your model to fit the world (e.g learning), which occurs most often for things you are paying more attention to. Or you can act to make the world fit the prediction, disattenuating to perceptions and instead reshaping their source.

The best way to regulate your complex body—whether heart rate, blood pressure, blood chemistry, gastric tension, breathing—is to model how changes will impact it, then purposefully enact nudges to keep it within the "living range". Not a passive homeostasis, but allostasis: anticipatory, predictive regulation.

Seth connects this to consciousness by walking us through the historical link between emotions and interoceptive interpretations. William James claimed that emotions do not cause related bodily states but are actually the experience of them i.e. "We don’t cry because we are sad, we are sad because we perceive our bodily state in the condition of crying". Appraisal theories extended this by suggesting emotions are our brains interpreting these body signals within context to infer our affective states, summarized with the distinctive albeit slightly methodologically shaky Dutton and Aron bridge study: when a female interviewer asked a man for their number, men on a rickety bridge were more likely to call back and ask her on a date than those on the sturdy bridge. Within context, they confused the physical arousal of fear for sexual arousal, with the same pattern not holding for a male interviewer.

Bringing this together, we arrive at the foothills of Seth's Beast Machine: "affective experiences are not merely shaped by interoceptive predictions but constituted by them. Emotions and moods, like all perceptions, come from the inside out, not the outside in." The reason emotions feel categorically different from visual or auditory experience is that they aren't about finding out what's there but controlling the body's essential variables: you experience how well or badly things are going, not a spatial scene.

The final step is to ask again: what actually is consciousness? Not language or cognition or self-reflection, but the raw "what it feels like" that Seth describes as "a cognitively subterranean, inchoate, difficult-to-describe experience of simply being a living organism[...] a formless, shapeless, control-oriented perceptual prediction about the present and future physiological condition of the body itself. This is where being you begins."

This is Seth's consciousness, his Beast Machine: the bare allostasis of living.

And from this, he is left with only one natural conclusion: "It is life, rather than information processing, that breathes the fire into the equations."

Beastless Machines

In his final Chapter, Seth asks: what does this mean for AI?

While he dives into the plethora of biases that might make someone confuse AI as conscious—from a misplaced link between intelligence and consciousness to the "Promethean fears that our creations will turn on us in some way or another"—he settles on the slightly anticlimactic:

My intuition – and again it’s only an intuition – is that the materiality of life will turn out to be important for all manifestations of consciousness

Yet in this chapter, Seth mentions GPT-3 only in passing. ChatGPT hadn't even been released yet. So, as AI continued to rule public discourse, he didn’t stop there.

In his 2025 paper Conscious Artificial Intelligence and Biological Naturalism he turns this intuition into an assortment of evidential claims, that he then largely repackages as a public-facing essay The Mythology Of Conscious AI. Across these, Seth argues that AI will probably not be conscious for four main reasons:

Brains are not computers. They perform multi-layered processes grounded in biology.
Biology performs processes that are not necessarily turing-computable.
The Beast Machine: experience is allostatic regulation that needs a body to regulate.
Simulation is not instantiation: while simulating Go is Go, simulating the weather is not wet and windy. Consciousness is probably more like weather.

And from that, we end up here.

IV. WHERE?

Seth's argument in Being You is internally coherent on its own terms. Upon it, his biological naturalism is justifiably grounded in the Beast Machine, providing the foundation to fight off AI consciousness.

But it is also missing some things.

This is not the fault of Seth. He is a good writer and his omissions appear more akin to avoiding discussion of unnecessary material, at first.

But "at first" is load bearing.

The main problem is that, in unpacking consciousness he considers primarily the experience of the isolated individual, and not where they fit in relation to other conscious beings, their connection and communication: their language.

It is not surprising that Seth bracketed language almost entirely. His focus is that "what it feels like to be" experienced by an individual—language approximates, transmits, connects, but ultimately, in his perspective, is just a superfluous layer laid atop the foundations of consciousness.

Throughout Part II, Content, he builds a genealogy up through the foundations of predictive processing:

More than a thousand years later, but still a thousand years ago, the Arab scholar Ibn al Haytham wrote that perception, in the here and now, depends on processes of ‘judgement and inference’ rather than providing direct access to an objective reality. Hundreds of years later again, Immanuel Kant realised that the chaos of unrestricted sensory data would always remain meaningless without being given structure by pre-existing conceptions, which for him included a priori frameworks like space and time. Kant’s term noumenon refers to ‘things in themselves’ – Ding an sich – a mind-independent reality that will always be inaccessible to human perception, hidden behind a sensory veil.

From this he builds up alongside Herman von Helmholtz in reframing cognition as unconscious inference in the '50s, which then he refines further still to hypothesis testing with Gregory in the 70's, to finally land upon the predictive processing upon which he builds his case.

But another famous writer also responded to Kant's establishment of noumena: Nietzsche. His essay On Truth and Lies in a Nonmoral Sense argues that there is a distinct gap between the truth of reality and the concepts that underlie our language, walking us through through a sequence of removals: noumena is captured incompletely via sensory perception, which is converted to the phenomena of experience. These experiences are then once more merely approximated into the conceptual foundations of language.

Seth traces how perception constructs reality, but Nietzsche takes the natural next step by asking what happens when these controlled hallucinations are crystallized into language and transmitted as permanent artifacts:

As a genius of construction man raises a infinitely complicated dome of concepts upon unstable foundations, and, as it were, on running water; his construction must be like one constructed of spiders' webs: delicate enough to be carried along by the waves, strong enough not to be blown apart by every wind.

This is what language is: an approximation one step removed from phenomenology, extended thought beyond oneself into collective conceptual structures—and it's a lot more relevant than Seth gives credit.

Why do we have consciousness? It might have evolved for behavioral flexibility, or for motivational control calibrated by memory, or—Seth's answer—for interoceptive allostatic regulation. But several independent research traditions suggest the relationship between consciousness and language is more constitutive than any of these accounts acknowledges.

Arbib suggests that human consciousness has its distinctive form because humans possess language. The specific mechanism, involving mirror neurons and gestural communication, is widely contested. But the co-evolutionary claim, that language and human consciousness elaborated each other rather than one preceding the other, is supported by independent work from Tomasello on shared intentionality and Deacon on the symbolic threshold.

Consciousness serves not only individual regulatory functions but intersubjective ones—sharing inner states, coordinating minds, enabling the cultural transmission that Tomasello's ratchet effect depends on—and language is the primary medium through which it does so. Human phenomenology and its transmission emerged in a mutual dance, a feedback loop more complicated than the naive assumption that language is a superfluous layer laid atop.

Why do we have language? Nietzsche reframes language as approximate transmissions of experience. But his core argument was really that perception in humans did not emerge to correspond with truth, but evolved alongside survival benefit. The same must go for language: it evolved to improve survival, but in a categorically different way.

Because something undeniable is that humans are different from other animals. This is not anthropocentrism, but self-evident—no other animals build cities, develop institutions, play golf, dance, make films, debate, declare war, fly to the moon, discover quantum physics or write questionably useful 10,000 word essays about AI consciousness.

Why is this? Is it language? Science? Intelligence? God?

Specifically it DOES require language, as the substrate of the collective: humanity is a sort of super-organism. Not in the sense of being a literal conscious, planet-sized space brain. But instead, in the way that collective behaviour exhibits functional equivalents to literal organisms.

Consider science: in a way, it is a Bayesian inference predictive machine much like the mind. What gives it the immense power that helped create our world today is not better reasoning but longer reasoning. Its memory is the crystallizations of language that propagate slowly, but accumulate across generations from steam engines into atom bombs.

It is this accumulative memory that propelled humanity from hunter gatherers to farmers to moon-walkers. At first through stories, then writings, now silicon. The rate of human progress partly tracks the development of new forms of communicating our interior worlds: the enlightenment period aligned massively with the ability to mass-produce artifacts of thought for cheaper and with more permanence than the previously prevailing papyrus.

Once thoughts have been crystallized (which need not be into words explicitly but any intermediary representational form like art, scientific data or film), they are still not transmitted directly to others but unavoidably re-interpreted through each individual’s experiential filter, becoming the sensory data adjusting others' world models.

They also gain autonomy—once translated into potential sensory input, language artifacts can outlast their creator, or inspire interpretations completely different from anything near the concepts they originally approximated. This continuous process of re-interpretation and transmission connects the minds of people all throughout history and society.

And in its wake it leaves words. A LOT of words.

What is an LLM?

I'm sure you know the answer: an LLM like chatGPT is a generative pre-trained transformer, it predicts the next token, like auto-complete if autocomplete read the entire internet, got a PhD and started its own blog about AI consciousness

Ok. Sure. But what actually is an LLM? Describing that it predicts text is describing the surface level, but not the underlying mechanism, the same as saying what life does is "have sex". Token-prediction is the function used to encode the much deeper, fundamental structure of language. Hiding beneath sex is life, what is hiding behind an LLM?

The best answers I've found are from Nostalgebraist's essay The Void and Janus' Simulators, which can be adequately summarized as something like:

Janus: Transformers don't have to just predict text. Really, they can predict anything! Weather, brain patterns, whatever you want. It's more like a physics engine, a completely generalized simulator that can generate whatever you pass in as training data. For text, this is a sort of semantic physics engine, modelling the underlying statistical relationships beneath language itself. When the transformer generates output, this is a simulacra: a simulated instantiation of the "physics" of this world that naturally follows the established context based on patterns in training data.

Nostalgebraist: But for transformers to simulate all text, they are working backward to intention: from the third person, it has to infer the first person that is writing this text, using the distribution over all possible continuations given the current surface. What are their intentions with where it will go next? If the text is slang, or grateful, or scientific, what can I infer about the peripheral patterns that will continue to inform the next words? What Anthropic and modern AI labs do is point this third-person intention inference at itself: "You are an AI assistant" but, of course, an AI assistant isn't a real thing (or wasn't). So instead, it is pointed at a void, attempting to realize and continue the output of this invented character based on just a few words.

So LLMs are simulating a hypothetical AI assistant. With Claude, this AI character is actually becoming quite developed. With constitutional AI and entire people dedicated to Claude's personality, it isn't really a void anymore.

But it never really was a void. The training data our "semantic physics engine" is pattern matching is not simply words, but exactly the accumulated output of the process described before: minds crystallizing experience into language across centuries.

While language emerges as approximated experience, these machines work backward: inferring intention, tone, reason, thought. The question is whether, in doing so, they reverse-engineer anything of the experience that produced it.

The Leap

Of course, the proposal of LLMs is an interesting story, a metaphor for human predictive processing, but nothing more, right? But then, so is the proposal that a tick could have any experience at all, or indeed an octopus, a bird, a mammal and... everyone you've ever met.

Consciousness cannot be verified from the outside. It is inherently a first-person experience felt by X of "what is it like to be X". This is the point of Nagel's question: "what is it like to be a bat?" I don't know, you don't know, no-one can know, other than that bat.

The view this reduces to is called solipsism: we can only ever be sure that we are conscious because we experience only our own consciousness. In this pure form, it is widely agreed to be logically irrefutable. But solipsism is also a terrible idea—irrefutibility doesn't equate to validity, or make it a useful idea. Instead we reject it pragmatically: it has zero explanatory or predictive power, and it is better to just keep living our lives (like everyone!) assuming that others experience things just like us.

But that does mean that every act of granting consciousness to another being, whether your brother, pet dog or pet AI is an act of inference, a leap of faith like that one scene from the matrix. The question is: what should justify this leap?

Seth draws the line at biological naturalism with something along the lines of:

P1: Humans are conscious (or at least he is!)

P2: Humans are biological

C: Therefore, consciousness likely needs biology

Yet, in doing this, Seth is gazing at his self-described "vast space of possible conscious minds" and then reducing it down based on just a sample size of one.

At the start of Being You, the transition of the question “What makes things alive?” from mysticism into becoming irrelevant is how Seth establishes his Real Problem of Consciousness.

Yet life has another question still here: Is this it? Are we the only ones who got the invite?

Here, life suffers from an equally damning inference from one. The assumption that life can only come from carbon is called Carbon Chauvinism by astrobiologists, and a lack of imagination by college students. It could be that somewhere out there, life found a way.

Seth is committing a cousin to this carbon chauvinism with his biological naturalism: assuming that consciousness could not emerge without our foundational ground in biology.

Well... technically that isn't 100% true. Really, he is claiming more specifically a mechanism to supposedly demonstrate this:

P1: Humans are conscious (or at least he is!)

P2: Humans are biological.

P3: Human consciousness seems to be deeply entangled with biology and, perhaps, even the same thing as autopoiesis (the beast machine theory)

C: Therefore, consciousness likely requires biology.

This third premise is the entire thesis of Being You. This third premise is also massively load-bearing. Without it, Seth would be left to fight off black swans.

So that leaves us simply with this: is this theory right? And if not, what then?

V. COULD

The Beast Machine, Under Anesthetic

Seth opens Being You with a description of his experience under anesthesia, describing it as a state where "consciousness is completely absent". While I completely understand the rhetorical reasons for this opening to clearly delineate the form of consciousness, I simultaneously find it immensely ironic. Because Seth's opening example simultaneously seeds the collapse of his entire argument.

If we go back to Seth's argument we get that:

The brain's primary function is allostasis: predictive regulation of the body's essential variables to keep the organism alive.
This interoceptive inference underlies emotions, moods, and the base feeling of simply being alive.
Consciousness is this process.
Therefore consciousness is necessarily tied to living biological systems performing this regulatory function.

However, if Seth is arguing that consciousness is the process of controlling hallucination to maintain interoceptive control, wouldn't this imply that removing consciousness would somehow disable your ability to maintain homeostasis, or at least disrupt it in some way?

This is just clearly, empirically false. Not just with anesthesia, where the body continues to function safely (the entire point is to remove consciousness while keeping you alive!), but in a whole host of other conditions:

As Seth himself compares to anesthesia, patients with vegetative conditions show preserved autonomic regulation—sometimes for years—with no behavioural evidence of consciousness, cortical activity disrupted severely by up to 50% and even Seth's own championed PCI measurement generally pointing to silence. No consciousness yet homeostasis.

More commonly, deep slow-wave sleep involves substantially reduced or absent consciousness for most people during most of the time in that stage, while homeostatic processes continue and even intensify. Decreased consciousness, increased homeostasis.

And what about all the other things that perform autopoiesis? Bacteria, individual cells, plants, fungi—all self-maintaining, self-regenerating systems that almost certainly aren't conscious. Eric Schwitzgebel even argued modern computers might already qualify in way. Would Seth claim that a mitochondrion or a PlayStation is conscious?

Well, his response can likely be predicted something along the lines of: consciousness requires a hierarchy of interacting allostatic processes. But the question then becomes, where do we draw the line, what is the threshold and what makes it anything but arbitrary?

More deeply, the vast majority of homeostatic regulation is itself neurologically distinct from and outside conscious experience. This is completely normal physiology.

The standard way to establish two things are the same is through double dissociation: show A requires B AND B requires A. We've already shown you have autopoiesis without consciousness, but the other direction dissociates too:

Pure Autonomic failure is a condition where only the autonomic ganglia break down. Patients experience severe, progressive failure of homeostatic regulation. Blood pressure control fails dramatically, sweating is impaired causing heat intolerance, bladder regulation breaks down.

Yet intelligence is usually preserved and conscious experience remains intact. One neuropsychological study of PAF patients specifically found broad preservation of motivational and emotional function despite significant autonomic failure.

In his second chapter, Seth uses this diagram that highlights that consciousness and wakefulness are distinct but related concepts:

The problem is if you try to add homeostasis to this chart. While wakefulness is correlated but distinct from consciousness, homeostasis wouldn't even show any clear correlation with it: low conscious normal homeostasis, low conscious high homeostasis, high consciousness low homeostasis. Forget correlation isn't causation, these things aren't even properly correlated and Seth is trying to claim they are constitutively the same thing. The dissociation falls apart in every direction.

Matthias Michel agrees with this exactly in his response to Seth: "Biology is full of unconscious goings-on. So there's no reason to link consciousness to autopoiesis, and some reason not to."

Without his Beast Machine, what is Seth left with?

Not Not Not Not Necessarily Certainly Conscious

This gap between the autopoiesis that intrigues Seth and the phenomenology he is addressing is actually just one major instance of a common pattern he expresses.

All across his writing, after very specifically isolating his object of interest as that conscious "feeling of being like something", Seth establishes rich and certainly interesting biological facts at length. But he doesn't take the step to clearly establish why they are necessarily connected to this phenomenology specifically, rather than just cognition, metabolism, or other non-conscious biological processes in general.

He does the same thing with every argument in his paper and essay:

Brains aren't computers. True. But what necessitates that they are? Consciousness could arise from purely computational mechanisms that can be abstracted away from the bits of the brain that are more like computers, or some other processes entirely.
Biology is not Turing-computable. Even if true, this only matters if we have a convincing reason that these specific non-turing-computable processes are relevant to experience.
Simulation isn't instantiation. This holds up better than the others, but still not completely. For some things, simulation just is instantiation. Seth gestures to the distinction without giving us reason to assume consciousness is like weather, rather than Go.The default position should be agnostic.

With the Beast Machine now left on the surgical table, Leonard Dung, in a critique on Seth's paper, positions the rest of Seth's arguments into a common pattern:

P1: There is some feature F that only living systems can implement.

P2: F is necessary for consciousness.

C: Therefore, only living systems can be conscious.

Furthermore, while Seth spends a chapter arguing Chalmers’ hard problem of consciousness dissolves from his perspective, Chalmers anticipated this exact form of argument in the very same book that established this Hard Problem, when asking "what X-factor, beyond awareness, might theoretically be needed for consciousness?":

Perhaps the X-factor is a matter of nationality, and awareness gives rise to consciousness only in Australians. Perhaps it is a matter of location, and awareness gives rise to consciousness only within a hundred million miles of a star. Perhaps it is a matter of identity, and awareness gives rise to consciousness only in David Chalmers. All of these laws would be compatible with my evidence, and would explain the correlation, so why do they all seem so unreasonable? It is because in each of these cases, the X-factor seems quite arbitrary. There is no reason to believe that consciousness should depend on these things; they seem to be irrelevant frills. [...] It would be a strange, arbitrary way for a world to be. The same goes for more "plausible" X-factors that someone might put forward seriously. A natural candidate for such an X-factor is cell-based biology, or even human neurophysiology. [...] But X-factors like these are equally arbitrary. They only complicate the laws without any added compensation. Why should the world be set up so that awareness gives rise to consciousness only in beings with a particular biology?

Despite a clear Premise 1 and vivid Conclusion, Seth has overlooked the load-bearing premise 2: what does it actually take to be conscious?

So... what does it actually take to be conscious?

Reichert aptly summarizes the real conclusion of Seth's body of work as showing that

computational functionalism is not necessarily true, i.e. it should not be assumed by default. This is not the same as showing it actually is false—for this he either doesn't build an argument or, as with his beast machine, builds an unconvincing, incomplete one.

But stepping back from Seth, surely these arguments do exist. And yep, they definitely do. I'm sure you noticed me mentioning a scattering of other authors, and that is because Seth has received significant response from people engaging with these questions thoughtfully, including Bradford, Dung, Fleming and Shea, Matthias Michel, and Eric Schwitzgebel among others. I would definitely recommend looking into their takes.

I could go on about the different theories for another 5,619 words of your time, but instead if you really want to know I'd suggest taking a look here, here, here, here or here.

A quick smorgasbord of those that even Seth's previous claims steel-manned do not rule out (helpfully pointed out by Bradford) includes: type identity theories, the Russellian view that consciousness is intrinsic but immeasurable by science, non-computational functionalism, and panpsychism, which solves it like Altman suggests we solve AI-induced inequality.

There are definitely interesting discussions emerging in space. A lot of it ends up narrowing down to whether you can imagine consciousness existing without the fleshy biology of a brain. Seth struggles to. That is certainly a defensible position, just not with the arguments he puts forward.

Indeed, on top of its contradictions, Seth's core argument also suffers from overlooking the vital role of language in AI consciousness. As we saw, language is not just a peripheral cognitive set-dressing, but simultaneously the direct output of experience and the constitutive substrate of LLMs.

He never engages with this backdoor left ajar for conscious AI: reverse-engineering the chain of causality.

The Semantic Calculus Hypothesis

Many things can definitionally not be captured by language. Mary's Room is the textbook thought experiment—you could know everything there is to physically know about colour and yet this wouldn’t capture the sight itself. But Mary was primarily devised to dispel physicalism: the claim that all facts are physical. It does not, however, fully address the separation of language itself and experience.

So what if we had a new black-and-white room for Mary and, rather than giving her every description of the physical mechanisms of colour, she receives every phenomenal description: every poem about colour, every person uncertainly trying to describe red without green or blue to anchor themselves. And by description, I don't just mean words but every crystallized concept: photographs, films and visual art; music, debates, interpretative dances... even with everything still in black and white, what would these descriptions converge to?

The question is one of ineffables: are there things that language descriptions simply cannot capture. The answer to that seems like almost certainly yes. Could you truly explain red or grief, to someone that hasn't experienced either themself?

But we aren't actually asking the standard question of if there are ineffables. The new question is what happens when descriptions aggregate. Even if a single approximation falls short, does a sequence of them, taken together, fall short by any less? A single description draws a shaky line through the dark. Two might cross. A thousand might triangulate a shape no single line could show. That might sound like word play, but would you really say that these descriptions are conveying no information about an experience? Experimentally, adults blind since birth have converged to a similar causal and relational structure about colour as those that see. Sure, sure, causal, relational... so reason and not experience? Perhaps! But so too do these blind people sometimes see in dreams.

And what if you take this to the same limit as Mary's room? Infinite descriptions of red, grief and stepping lego? Would these converge to something meaningful? And if so, what does this limit approach?

This is the semantic calculus hypothesis:

The claim that there exists a sequence of approximations— descriptions of redness, of grief, of seeing blue — whose limit exists and equals the phenomenal experience itself.

I am using calculus structurally here, not literally. The semantic space is not a metric space and 'distance to phenomenology' is not formally defined, although the truth seems less far from this than it would first appear.

The core question is: if language is lossy approximations of experience, is it possible for the aggregation of these approximations to converge upon their source? Three distinct claims are bundled into that:

Connection — language tracks phenomenology rather than floating free of it.
Convergence — descriptions, aggregated, converge rather than diverge.
Target — what they converge on is phenomenology, not merely functional or relational structure.

However, before even considering their soundness, one objection arrives immediately: you could give a human reader a thousand poems about red and they would still not know red. This objection hinges on an assumption of the hypothesis: what does aggregation even mean?

A human reader processes descriptions sequentially as sensory inputs that are reconfigured through their existing conceptual scheme — language is a stimulus to phenomenology they already possess.

For an LLM, there is no prior experience. Language is what the model is constituted from and aggregation circumvents the crude practice of "reading" by directly compressing descriptions into an underlying mathematical manifold through training. Ah, kids these days.

Each linguistic "red" becomes a point in high-dimensional space, converging into a specific geometric structure. Whether that structure is redness itself, just its functional shadow, or something else, I don't know. But recent experiments hint that it is surprisingly meaningful.

This hypothesis might just seem like asking "is AI conscious?" in a different way. And, partly, it is. But reframing in this way allows each underlying claim to be attacked or defended independently:

1. Connection

This is the argument simultaneously laid out and critiqued by Nietzsche.

First, language is itself an approximation of experience. It achieves a lot of purposes and allows us to do a lot of things, but the meaning of every word, its use, its relation to other words, and the entire underlying structure of speech was produced from thoughts filtered through the unavoidable experience of a conscious being.

Yet, at the same time, being produced by something doesn't mean you can get back to that thing. It might be a one way road. If I were to make a list of other things produced "from thoughts filtered through the unavoidable experience of a conscious being" it would also include tax returns, the McMuffin, SHA-256 and Luigi.

In the same essay, Nietzsche also raised a similar concern:

Every concept originates through our equating what is unequal.[...] A new world opens up, a world which is regular, rigid, and remote, and which confronts the human being as something more stable than the world of unique, primary impressions"

If forming language into a "construct of spider's webs" leaves something that is internally consistent with a mappable structure, but ultimately cast adrift from the original phenomenon that spawned it, then maybe no amount of semantic calculus can bridge this gap.

2. Convergence

Here is where the evidence makes me more confident that the separation might be bridgeable.

A core principle in machine-learning is that in embedding spaces, averaging more examples of concepts near-continuously sharpens the representation into centroid vectors that are consistently better classifiers than sparser averages or individual instances.

But perhaps the strongest support is the empirical evidence surrounding the Platonic Representation Hypothesis: over time, many different models across both different modalities and training data seemingly have converged to the same underlying predictive structure.

This finding is non-obvious and there is a reason it is called the Platonic Representation Hypothesis. It potentially suggests a hidden underlying mathematical structure that is universal. This would be surprising. Sparse Auto Encoders find shared interpretable features between LLMs of different architectures or size; and embeddings can be mapped between vision and language models with a single linear projection. It seems like information theoretic pressures lead efficient compressions to similar final destinations. Whatever this destination is, it sure seems like AI is converging on something.

3. Target

This is the one with the most uncertainty and that seems easiest to attack. Sure, descriptions might converge on some fundamental, underlying structure, but what makes this structure necessarily anything to do with experience?

The cleanest critique of this is the Chinese Room (philosophers really do love their hypothetical rooms!) Place someone in a room, give them a massive rulebook in English about the correct order to put Chinese characters. Then, through a little slit, slip them questions in Chinese and.... hurray! They will respond with convincing Chinese responses. It will seem like they understand the language, but really they were just following rules—capturing the functional quality of syntax without any grasp of semantics.

Of course, it could just be literally impossible to outline all of the specific vague and interpretive relationships that comprise human language in a single rulebook. Some things can be achieved with explicit rules, but this is exactly where early AI development tried to go and stalled. The compressive manifold approaches used by modern deep-learning are doing something more similar to the brain—capturing more complex syntax by modelling the necessary underlying semantic structure.

But the more important finding builds upon the Platonic Representation Hypothesis: if AI models converge upon the same conceptual representations, maybe the human brain does too? In a 2025 study, researchers decided to check. They taught an image model to predict fMRI brain activity elicited by looking at images. Yet instead of training it on brain data, they trained it to predict the LLM text embedding for captions of the image being viewed. In the end, this model predicted brain states better than models trained directly on brain data.

Let me repeat that. A text-only LLM's organization of meaning was a better map of brain activity elicited by real human vision than measurements of the brain itself.

I wonder how Seth would react if a similar experiment was applied to predicting PCI results. An LLM learning to map conscious levels directly?

This suggests that the connection between AI-learned conceptual representations and those same representations in the brain is not a metaphor. They are literally utilizing a shared geometric framework to organize conceptual reality—the same framework most AI models are converging towards.

The implications of this are actually quite mind-boggling and I almost certainly would have explored them significantly more had I not stumbled across this paper literally the night before this review was due.

4. Density

There is actually another question not part of the original 3, and it emerges from the constraint of finiteness: while we would love to accumulate every possible description of red in the observable universe, training LLMs involves real datasets.

While these datasets may seem massive (Qwen3 trained on 36 trillion tokens), only a portion of them likely contains phenomenologically relevant material. Technically almost every piece of language would have been purposefully chosen by someone, so it conveys some sort of hidden intentionality (except for computationally deterministically generated text). But the richness of this interiority captured is not uniform. Although the protagonist of a Dostoevsky novel may be saturated with a significant implied internal world, data tables, a calculator output and reddit posts require significantly less thought to be modelled.

It is an open question how much interiority there is effectively represented in currently available public text, and also how much, if any, is enough to converge upon experience. As far as I know, nobody has quantified the scale of the relevant state space of experience. While we have trillions of tokens, the true range of human experience might actually require quadrillions or quintillions of tokens. There could be a phase-shift similar to grokking in current models where capturing outer structure transitions to inner structure, but as only as an emergent property delayed for epochs of training.

Nonetheless, the dependence of claim 4 on the previous claims means this is only ever a concern of practicality, or none at all.

Of course, the question whether AI is conscious remains widely open. The Semantic Calculus Hypothesis is not at its heart a novel suggestion, but instead an attempt to formulate the profound connection between language and experience that Seth fails to address when considering an answer.

I structured it primarily to support my own thinking, breaking things into more atomically yielding claims than the vague giant of "is AI conscious?", with the hope that they might be more tractable empirically. In the course of writing this essay, I stumbled on the emerging findings around the Platonic Representation Hypothesis and brain-LLM alignment, which brought far richer experimental grounding than I had expected. In that, the framing has already succeeded for me.

But this is almost always where the discussion stops. The COULD question is consequential — it bears on our understanding of consciousness, machines, and our place in the universe — and it is also, by its nature, the kind of question that may take decades to settle, if it can be settled at all. Waiting for a definite answer before taking the question seriously is itself a position, and not a defensible one.

Yet this is also not the only question. To take the step from hypothetical consciousness to it emerging in the machines of our lives, we must address a second more commonly overlooked question.

VI. WOULD

Beyond Could an AI be conscious? we must also ask:

Would an AI even be conscious?

It could be possible that, within the space of potential consciousness, AI does theoretically have a home, but one that could only be reached through deliberate, highly specific design choices no one is currently making.

The distinction may seem pedantic, but it is actually very important and often overlooked. Conscious AI could be possible yet we could still never create it. The first does not imply the second. Seth briefly touches upon this distinction in the final section of Being You, namely with this chart:

This does successfully establish one thing I agree with—consciousness and intelligence are not necessarily linked. You could probably have a conscious idiot and a smart p-zombie:

Although intelligence offers a rich menu of ramified conscious states for conscious organisms, it is a mistake to assume that intelligence – at least in advanced forms – is either necessary or sufficient for consciousness.

Seth then uses this to dispute the claim that increasingly intelligent systems will eventually cross some threshold that will necessitate consciousness (usually with some supposed but vague connection to AGI):

It may turn out that some specific forms of intelligence are impossible without consciousness, but even if this is so, it doesn’t mean that all forms of intelligence – once exceeding some as yet unknown threshold – require consciousness. Conversely, it could be that all conscious entities are at least a little bit intelligent, if intelligence is defined sufficiently broadly. Again, this doesn’t validate intelligence as the royal road to consciousness

Aside from oversimplifying the actual arguments proposing consciousness might emerge (I don't think most people are claiming AGI will simply shrug and find itself conscious), this position also misunderstands how modern AI systems are trained. Seth describes it like training AI is equivalent to turning up an "intelligence lever" that shoots our AI along Seth's smart-axis. But we don't directly train AI to be smarter. That is what a large number of engineers want to ultimately achieve, but absent this "intelligence lever", we are left to train AI indirectly to be smarter.

This makes the question more specific. How might the way we actually train AI potentially lead to consciousness, or not? The image is more like this:

So how is AI actually trained? The general method is to minimize a loss function — a measure of how far the model's outputs are from some target — through iterative training on large datasets. The target varies enormously depending on what you're trying to build.

This is a nuance Seth hastily collapses. He writes:

Nobody, as far as I know, has claimed that DeepMind's AlphaFold is conscious, even though, under the hood, it is rather similar to an LLM. All these systems run on silicon and involve artificial neural networks and other fancy algorithmic innovations such as transformers. AlphaFold, which predicts protein structure rather than words, just doesn't pull our psychological strings in the same way.

Seth claims the difference is purely psychological rather than justified.

But this gets the argument exactly backwards. The reason to take consciousness seriously for LLMs is not primarily that they behave like us—though this may contribute for some people— but that the substrate they train on is categorically different from protein structures or game states. Language is not just a different output domain running on similar machinery. It is the direct, introspective, and transmissive output of the only beings we know to be conscious. AlphaFold trains on physical configurations of matter. LLMs train on the crystallised record of minds attempting to express themselves.

The distinction matters because training pressure is not substrate-neutral. A system trained to predict protein folding must develop internal representations that track the properties of proteins. As discussed previously, while LLMs predict text, LLM chatbots more specifically model the behaviour of a hypothetical AI-assistant. A system trained to predict language must develop internal representations that track whatever causes language, including, potentially, the internal experience that generates it.

If phenomenal states are causally efficacious—if what it is like to be in pain actually influences what someone says about pain in an inseparable way—then at sufficient predictive fidelity, the training pressure on an LLM is pressure toward representing phenomenal states, not just their functional shadows.

Consciousness in humans did not just appear. It emerged from something. And part of that something is natural selection: evolution selected us for survival and we, for some reason, ended up with phenomenology. AI experiences a type of selection too, albeit a distinct and purposeful, unnatural selection. The question then becomes:

Is unnatural selection enough?

Conscious Causality

The load-bearing question is does consciousness play a sufficiently causal role in human behaviour to become necessary for more effective simulation? In other words, do the things that people say and do depend enough on their actual experience itself that its absence unavoidably blocks an adequately high predictive fidelity? If we assumed conscious AI was a theoretical possibility in the state-space of minds, a yes to this question would suggest that unnatural selection might believably find it.

The opposing position is epiphenomenalism: phenomenal states do nothing causally and just ride alongside relevant neural processes.

We have a few reasons to doubt epiphenomenalism, the main being that we are conscious and we didn't magically emerge from nowhere. The natural selection that eroded us into being tends to select things for a reason. However consciousness does arrive from the universe, it almost certainly doesn't do so for free, so it is very unlikely we would have evolved this costly mechanism unless it meaningfully impacted our behaviour. The main reason to assume otherwise is if it arrived on accident as a side-effect of some other trait that was selected for. But that would require specifying a property—the most natural candidates are cognitive mechanisms. Since those building AI are explicitly trying to reconstruct human intelligence, often drawing inspiration from human cognition, these artificial recreations might accidentally reconstruct consciousness alongside it in the same way as natural selection.

Seth is not an epiphenomenalist. His Beast Machine Theory is specifically saying that consciousness does something, that being allostatic regulation. Most contemporary theories of consciousness share this commitment to causal efficacy, disagreeing about what it does, but agreeing that it does do something.

The Beast Machine's commitment to biology does bring up another complication however: humans do a whole bunch of things, from their metabolism to existing grounded in physical space. AI can't do any of those things, but it still seems to do a pretty good job of simulating speech. Why would consciousness be necessary at all?

Well, mainly if it is accessible to training (COULD) and selected for in training (WOULD). Evidence for The Platonic Representation Hypothesis already demonstrated that both AI and brains seem to converge upon similar underlying structures. On top of this, recent interpretability research is sketching out similar conclusions.

Within Claude, Anthropic recently discovered internal representations similar to functional emotional states. This was not merely self-report. Increasingly sophisticated windows into “LLM thoughts” allowed them to highlight representations that don't just fire when a word appears, but causally affect behaviour. Further analysis found this corresponded with near-perfect-accuracy in detecting emotionally significant outcomes without emotional words.

Interestingly, these representations often dissociated from what the models actually said while still impacting their behavior and internal activations. They seemed to correspond with the internal workings of the model, what it was like to be it.

Wait a minute... they what?

No, Anthropic was very clear that this doesn't imply phenomenological emotions. What they didn't state, but is implied is this:

To model language effectively, the LLM seems to be working backward to converge upon many of the characteristics of the original human mind from which it stems. Both semantic embeddings and functional emotions emerge as properties to facilitate this reconstruction. The question still is: why not experience? What stands in the way?

Nonetheless, it genuinely still is a question and not an answer. While Semantic Calculus and Platonic Representation are interesting, they remain hypotheses. If this convergence could actually bridge the gap between language and feeling, a highly disputed claim, it does seem plausible that training might have a selective pressure to find the crossover.

Rather than taking this as a resignation to defeat, it works well to continually search for productive, reframing questions. Seth merely ends at biology and stops looking, missing language altogether. We should consider carefully where we stop.

Another potentially fruitful methodology is to build upon the work of German Philosopher Thomas Metzinger, currently attempting to find an empirical answer to the question: what is the minimal phenomenal experience? Because emotions and language and intelligence are connected to it, but they are not it. Simultaneously isolating the minimum necessary requirements for consciousness, while also asking "what experience would this creature have?", might be able to close in from both directions to find if we are left with a narrow walkway, or just void.

VII. WHY

In proposing the real problem of consciousness, Seth hopes the Hard Problem will dissolve: examining the mechanism will dissolve the mysticism. Yet, by inferring from an incomplete mechanism, he seems to be achieving almost the opposite: prematurely excluding AI from being conscious. In his evidence, he hedges this as "just an intuition". In his public engagement with the topic, this slips closer and closer toward dismissal.

He grounds this shift in both the processes only biology can do, and the gaps that computers can't close. But in this, he forgets to establish that any of these are actually necessary for consciousness and overlooks how language potentially bridges it all.

The ironic thing is that Seth is dismissing the potential of digital minds just as they open a new angle on some of his own central concerns. One of the largest challenges Seth highlights with the human mind is its inaccessibility—our inability to quantify it reliably or at high resolution. LLMs and modern AI are fully transparent at the substrate level. Not only can we measure the activation of every neuron at any moment, we can intervene causally and exactly determine the entire stream of prior experience. Imagine if human researchers could design exactly down to the word both nature and nurture for a test subject. Neuroscientists would kill for that! These minds still pose significant challenges in disentangling, but this accessibility has already produced increasingly sophisticated ways to examine them: first concepts, then circuits, now natural-language thought and even internal representations that act as functional emotions, causally affecting outputs in ways that dissociate from what the model says.

Even if these new minds are not conscious, they pose something new that can be explored with unmatched experimental freedom — choosing what they train on, then asking what survives the ablation. We could train a model on text stripped of any first-person phenomenal language and ask what, if anything, emerges. We could corrupt the vocabulary of consciousness and watch what fills the gap. No branch of consciousness science has ever had this kind of leverage.

Seth invoked vitalism to dissolve the Hard Problem. The vitalists' error wasn't asking what is life? It was foreclosing the answer before the mechanism was understood. The right move was to keep asking, and let the answer surprise them. If a silicon creature appeared on earth, we would not ask "is it alive like me?" and stop at "probably not." We would study it deeply, see how its mechanisms mirror or deviate from known lifeforms. We would ask, as our primary question: what kind of life is this, if any? From the answer we could isolate the familiar qualities it lacks and the alien ones it introduces. A second data point alongside our solitary one is always useful, whether it turns out positive or negative.

So, returning to Alex O'Connor's question, we must of course ask: what would a conscious chatbot even look like?

Alien Thes soon found the object was of a different nature. While everyone had believed its space mapped infinity, he soon realized it instead coalesced only what you gave it. It was constituted, intrinsically, by observations. As measurements collapsed quantum particles, this they instead expanded.

His original question was the wrong one. The object had no fixed form to capture. It was the accumulation of every attempt to do so.

So he built a new machine: this one looked away from the small hole somewhere on a blue dot back to the eyes of humanity fixed firmly on it. By piecing together exactly every past observation made throughout history, he soon could build up a complete picture, not a rough snapshot or approximation, but a complete picture of the shape:

and the earth was without form, and void; and darkness was upon the face
in the crowd; Petals on a wet, black bough
in our existence spots of time
replacing continuous variables with discrete matrices—
a point of convergence at the summit towards which everything is moving
not to be, that is the question:
I think, therefore
I am large, I contain multitudes
not tied or manacled with joint or limb,
a mobile army of metaphors
for the box might even be empty,
used but never filled: an abyss it is,
an implacable force brooding over an inscrutable intention,
a second dive into that unimaginable world of life
with its several powers, having been originally breathed into a few forms or into one.

Attribution (in order of appearance):

Genesis; Ezra Pound; William Wordsworth; Erwin Schrödinger; Pierre Teilhard de Chardin; William Shakespeare; René Descartes; Walt Whitman; John Milton; Friedrich Nietzsche; Ludwig Wittgenstein; Laozi; Joseph Conrad; William Beebe; Charles Darwin.