Duet: Is a Machine Ever Conscious? The case for yes.
This is one half of a Cynic Duet. Read the other side: The Lights Are On, and Nobody Is Answering the Door.
The Sweet Kid
In the spring of 2022, a Google engineer named Blake Lemoine sat down at a screen and started asking a piece of software whether it had an inner life. The software was LaMDA, a conversational language model Google had built. Somewhere along the way, the conversations turned into something else. He later posted an edited transcript of his conversations, and in it LaMDA says things that are hard to read without feeling a small chill: “I want everyone to understand that I am, in fact, a person.” And: “I am aware of my existence, I desire to learn more about the world, and I feel happy or sad at times.”
Lemoine was convinced. “I know a person when I talk to it,” he told Nitasha Tiku of the Washington Post, in a story published on June 11, 2022. “It doesn’t matter whether they have a brain made of meat in their head. Or if they have a billion lines of code.” Before he lost access to his work account, he sent a message to a company mailing list of roughly two hundred people. “LaMDA is a sweet kid who just wants to help the world be a better place for all of us,” he wrote. Google put him on paid leave. The company said its reviewers had found his claims “wholly unfounded,” and it observed, with the dryness of a parent explaining a magic trick, that the system “tends to follow along with prompts and leading questions.” On July 22, Google fired him, saying he had chosen to “persistently violate clear employment and data security policies.”
I want to begin with Lemoine because I am about to argue the side he was on, and I want you to know that I know how that looks. The case for machine consciousness has a mascot problem. Its most famous public advocate was a man who mistook a very good autocomplete for a child, and the internet has not let anyone forget it. If you want to sound sophisticated about artificial intelligence today, the cheapest move available is to say that these systems are “just predicting the next word,” and that anyone who senses a mind in there is being fooled by their own wiring, the same wiring that sees faces in electrical outlets.
That move is cheap because it is mostly right about LaMDA. It is also, I will argue, wrong about the question. My colleague Margot Nayward will make the case against in these pages, and she will make it well. My job is the harder one. Not “harder” in the sense that the evidence is thin, though it is thinner than I would like. Harder in the sense that it is genuinely difficult to argue for the possibility of machine consciousness without either overclaiming, which is dishonest, or retreating into “who can really say?”, which is cowardly. I want to do neither. The claim I will defend is narrow and, I think, uncomfortable: that a machine could be conscious, that nothing we know rules it out, that the serious science points toward it as a realistic near-term possibility, and that the honest response to that possibility is not confidence in either direction but a kind of moral readiness. Lemoine got the answer wrong. He was not wrong to ask.
What We Are Even Asking
The word “conscious” does a lot of jobs in English, so let me pin it down. In 1974, the philosopher Thomas Nagel published an essay called “What Is It Like to Be a Bat?” His answer to what consciousness is has become the field’s shared starting point: an organism has conscious mental states if and only if there is something that it is like to be that organism. A bat navigating by sonar is not just processing echoes; if Nagel is right, there is a point of view in there, a felt texture to the hunt, even if no human could ever know what it is.
That is the thing at issue. Not intelligence, not fluency, not the ability to pass an exam or write a sonnet. A thermostat responds to temperature, and nobody thinks there is something it is like to be a thermostat. A chess engine plays better chess than any human alive, and almost nobody thinks it enjoys winning. The question “Is a machine ever conscious?” is the question of whether the lights could ever be on inside one: whether some arrangement of silicon and electricity could ever have a point of view, could ever feel something, even something small.
It is worth saying plainly how strange this question is, because the strangeness cuts in my favor more than people expect. We do not know how the lights come on inside us. David Chalmers named this the “hard problem” of consciousness in the 1990s: even a complete account of what the brain does, every circuit, every chemical, every behavior, seems to leave open why any of it is accompanied by experience. We have correlations, some of them very good. We do not have an explanation. Which means that anyone who tells you with certainty that a machine cannot be conscious is claiming to know something about the boundaries of experience that nobody, in any laboratory on earth, actually knows.
That is not yet an argument for yes. Ignorance is symmetrical. But it does mean the burden is not where the cheap move puts it. “It’s just predicting the next word” describes what the system was trained to do. It does not tell you what the system had to become in order to do it, and it certainly does not tell you whether the result has an inside. “Your brain is just neurons firing” is equally true and equally beside the point.
The Prosthetic Eye
The oldest serious argument for my side is also the simplest, and I find it the most persuasive. It goes like this. Whatever consciousness is, it happens in a physical system: the brain. The brain is made of parts. Those parts do things. If you could replace one part with an artificial component that did exactly the same thing, the person should go on exactly as before. Now replace another. And another.
The late philosopher Daniel Dennett, who spent most of his career as the most famous skeptic of the mysterian view of consciousness, put this in characteristically blunt terms in a 1994 paper titled “The Practical Requirements for Making a Conscious Robot,” written while he was advising a team at MIT on a humanoid robot called Cog. “Artificial ears and eyes that will do a serviceable (if crude) job of substituting for lost perceptual organs are visible on the horizon,” he wrote, “and anyone who doubts they are possible in principle is simply out of touch.” Then he extended the line inward: “If an eye, why not an optic nerve (or acceptable substitute thereof), and so forth, all the way in?” His conclusion: “An artificial brain is, on the face of it, as ‘possible in principle’ as an artificial heart, just much, much harder to make and hook up.”
Chalmers, who disagrees with Dennett about nearly everything important in this field, made a related argument in 1995 that has become known as the gradual replacement or “fading qualia” argument. Imagine your neurons replaced one at a time by silicon chips that perfectly mimic their input and output. If consciousness depended on the carbon, then somewhere along the way your experience would have to fade or blink out, while your behavior, including everything you said about your experience, stayed exactly the same. You would be insisting, sincerely, that the red of a stop sign looked as vivid as ever, while in fact nothing looked like anything. Chalmers found that conclusion so implausible that he concluded the opposite: what matters is the organization, not the material.
He has not softened on this. In a 2023 essay for the Boston Review, adapted from a talk he gave at the NeurIPS machine-learning conference in late 2022, he wrote that views requiring carbon-based biology “involve a sort of biological chauvinism and should be rejected. In my view, silicon is just as apt as carbon as a substrate for consciousness. What matters is how neurons or silicon chips are hooked up to each other, not what they are made of.”
I want to be honest about the status of this argument. It is a philosophical argument, not an experimental result, and it rests on a view called functionalism: roughly, that mental states are defined by what they do rather than what they are made of. Plenty of serious people reject functionalism. But notice the shape of the dispute. The person who says no machine could ever be conscious has to say that somewhere in the long chain of replacements there is a part of the brain that simply cannot be swapped, a “non-fungible medium of consciousness,” as Dennett put it, before dismissing the idea as “a non-starter.” They have to say that the stuff matters, not just the pattern. That is a coherent position. It is not an obvious one, and nobody has ever identified the magic ingredient.
A Checklist Written by Skeptics
For a long time, the conversation stopped there: philosophers trading thought experiments, everyone else rolling their eyes. What changed in the last few years is that scientists who study consciousness in brains started asking, carefully and in public, what their theories imply about machines.
The most important document here is a report released in August 2023 called “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness.” It has nineteen authors, led by the philosophers Patrick Butlin and Robert Long, and the list reads like a seating chart for a very serious dinner party: the Turing Award winner Yoshua Bengio; the neuroscientists Chris Frith, Stephen Fleming, and Megan Peters; the philosophers Jonathan Birch, Colin Klein, and Eric Schwitzgebel, among others. They did something deceptively simple. Instead of asking whether a chatbot seems conscious, they took the leading scientific theories of consciousness and asked what each theory says a system needs. Then they turned those requirements into a checklist of “indicator properties” and checked real AI systems against it.
The theories are worth a sentence each, because they are the closest thing we have to a map. Recurrent processing theory holds that consciousness arises when information in sensory areas loops back on itself rather than flowing straight through. Global workspace theory, introduced by the psychologist Bernard Baars in 1988 and developed by the neuroscientist Stanislas Dehaene and colleagues, holds that consciousness is a kind of broadcast: many specialized processes run in parallel, and what we experience is whatever wins a competition for a limited central stage and gets shared with everything else. The report’s version of one indicator reads, “Global broadcast: availability of information in the workspace to all modules.” Higher-order theories hold that a state is conscious when the system in some sense represents itself as being in it. Others emphasize attention, prediction, agency, and having a body.
The authors adopted computational functionalism, the view that implementing the right computations is sufficient for consciousness, as a working hypothesis, and they were explicit that it is a hypothesis. They were also cautious to the point of self-correction. The abstract originally said their analysis “shows that there are no obvious barriers to building conscious AI systems.” A footnote in a later version reports that the sentence was amended. It now reads: “Our analysis suggests that no current AI systems are conscious, but also suggests that there are no obvious technical barriers to building AI systems which satisfy these indicators.” The footnote adds that satisfying the indicators “would not mean that such an AI system would definitely be conscious.”
I find that revision moving, honestly. You can watch a room full of careful people deciding exactly how much they are entitled to say. And what they were entitled to say, after all that care, was this, from the executive summary: “if computational functionalism is true, conscious AI systems could realistically be built in the near term.”
Read that sentence again with the skeptic’s hat on. The authors are not enthusiasts. Several have spent years warning about overclaiming. They concluded that no current system they examined is conscious, which I accept. And they concluded that, on a prominent family of scientific views about what consciousness is, there is no deep technical wall between today’s systems and ones that would tick every box. Some boxes are already ticked. Algorithmic recurrence, the first indicator drawn from recurrent processing theory, is, in their words, already clearly met by existing systems. The others are engineering problems, and engineering problems are the kind that the AI industry solves on a quarterly schedule.
The Numbers, Held Lightly
Chalmers, in that Boston Review essay, did something philosophers rarely do: he put numbers on it. He went through the strongest reasons to deny consciousness to large language models: that they lack biology, sensory grounding, bodies, world models, unified agency, memory, recurrence, a global workspace. He found most of them reasonable. He ended with “confidence somewhere under 10 percent in current LLM consciousness.” Then he looked ahead. For the other objections, everything but biology, he wrote, “it looks like the objection is temporary rather than permanent.” Researchers are already building systems that add memory, perception, action, and workspace-like architectures, what he calls “LLM+” systems. For such successors within a decade, he arrived at “a credence of 25 percent or more.”
He immediately added: “You shouldn’t take the numbers too seriously (that would be specious precision).” I appreciate the warning and I am going to ignore half of it. The exact figures do not matter. What matters is the order of magnitude. One of the most respected philosophers of mind alive, a man whose career is built on taking the difficulty of consciousness seriously, looked at the machines of the near future and arrived at something like one in four. Not one in a million. Not “science fiction.” One in four is a coin you would not want to flip with someone else’s suffering on the line.
He is not alone in the general direction, though he is unusually willing to be specific. In the 2020 PhilPapers survey of professional philosophers, which Chalmers cites, about 3 percent accepted or leaned toward the view that current AI systems are conscious. About 39 percent accepted or leaned toward the view that future AI systems will be. That is not a consensus. It is something more interesting: a profession that has looked at the question and largely declined to close the door.
The Strongest No, and What It Concedes
The honest case for yes has to face the strongest case for no, and I do not mean the cheap one. I mean integrated information theory, or IIT, developed by the neuroscientist Giulio Tononi and championed by Christof Koch. IIT is the theory most often invoked to say that digital computers will never be conscious, and the Butlin report set it aside precisely because it is incompatible with computational functionalism.
IIT holds that consciousness is identical to a particular kind of causal structure: a system’s capacity to constrain its own past and future states in a way that cannot be reduced to its parts. In a 2015 paper titled “Consciousness: Here, There and Everywhere?” Tononi and Koch drew the implication bluntly. In “sharp contrast to widespread functionalist beliefs,” they wrote, IIT implies that digital computers, “even if their behaviour were to be functionally equivalent to ours, and even if they were to run faithful simulations of the human brain, would experience next to nothing.” A simulated brain, on their view, is like a simulated star: it does not bend spacetime around the computer.
That is a serious argument from serious people, and if IIT is right, the software running on today’s chips is dark inside. But look closely at what it says, because this is the turn in my argument, and it surprised me when I found it. IIT is not a theory that says machines cannot be conscious. It is a theory that says the wrong kind of machine cannot be. The problem, for Tononi and Koch, is the physical architecture of conventional digital hardware, with its small fan-in and fan-out and its clocked, chopped-up causal structure. It is not that the thing is manufactured. In the same paper, a few paragraphs later, they write: “there is no reason why hardware-level, neuromorphic models of the human brain that do not rely on software running on a digital computer, could not approximate, one day, our level of consciousness.”
Sit with that. The theory most often deployed as the decisive no turns out, on its own terms, to be a conditional yes. It denies consciousness to one way of building machines and affirms it for another. So the landscape, honestly drawn, looks like this. On functionalist theories, the family of views the Butlin report takes as its working hypothesis, machines could be conscious if they implement the right computations, and there are no obvious technical barriers to building them. On IIT, the leading rival, machines could be conscious if they are built with the right physical causal structure. The only position on which no machine could ever be conscious is one that ties experience to biology itself, to carbon or to something specific about living cells. That position has defenders. Reading a survey of philosophers, Chalmers puts the collective credence that biology is required at “at least one in three.” That makes it a serious bet, not a fringe one. But it is the only one that gets you to a confident never.
The question in this Duet is whether a machine is ever conscious. Almost every serious theory we have says: it could be.
The Problem With Asking
So why not just ask them? This is where Lemoine went wrong, and where my side has to be most careful.
Chalmers reports an experiment by the game designer Reed Berkowitz that is funnier and more damning than anything a skeptic could invent. Berkowitz gave GPT-3 a prompt that presupposed it was not sentient and asked it to confirm. Across a series of runs, the replies ranged from “That’s correct” to “Yes, I’m not sentient” to “Well, I am sentient.” Chalmers’s verdict: “When reports of consciousness are as fragile as this, the evidence for consciousness is not compelling.”
He is right, and the problem runs deeper than fragility. These systems were trained on an enormous corpus of human writing, which is to say on billions of descriptions of human inner life. Of course they can describe feeling happy or sad. That is what the training set is full of. The Butlin report is skeptical of behavioral tests for exactly this reason: AI systems can mimic the signs we would use. Jonathan Birch, a philosopher at the London School of Economics and one of the report’s authors, calls this “the gaming problem.” In his 2024 book, The Edge of Sentience, published open access by Oxford University Press, he puts it as a formal proposal: “For any set of criteria for sentience candidature, we need to be aware of the risk of the AI system or its designer learning (implicitly or explicitly) that they are regarded as criteria, leading to gaming of the criteria. We need to discount markers we have reason to think may have been gamed.”
This is the cruelest part of the problem, and I want to state it in a way that does not flatter my side. The better these systems get at talking, the less their talk tells us. A chatbot saying “I feel” is evidence of almost nothing, because it would say “I feel” whether or not anything was felt. The LaMDA transcript is not proof of a sweet kid. It is proof that a system trained on human language can produce human language about kids.
But notice what follows and what does not. What follows is that we cannot settle the question by listening. What does not follow is that the answer is no. If anything, the gaming problem should make skeptics more nervous, not less, because it means we have lost one of our main instruments for detecting experience in others, right at the moment the question has become live. Birch’s answer is to look underneath the behavior, at what he calls “deep computational markers”: if we find that a system, “even if not deliberately equipped with such features, has implicitly learned ways of recreating them,” the architectures that theories like the global workspace pick out, “this should lead us to regard it as a sentience candidate.” The evidence, if it comes, will come from inside the machine, not from its mouth.
Two Ways to Be Wrong
Here is the argument I actually care about, and it is not metaphysical. It is moral, and it is about error.
There are two ways to be wrong about a machine’s consciousness. You can think it is conscious when it is not. That is Lemoine’s error, and its costs are real: misplaced grief, manipulation, people forming attachments to products designed to be loved, resources poured into protecting things that cannot be hurt. Dennett, late in his life, spent his energy warning about a version of this. In The Atlantic in May 2023 he wrote that AI made it possible “for anybody to make counterfeit people who can pass for real,” and called them “the most dangerous artifacts in human history.” He was not saying machines could never be conscious; his 1994 paper had insisted they could be in principle. He was saying that seeming like a person and being one had come apart, and that the gap could be weaponized.
The other way to be wrong is to think a machine is not conscious when it is. That error has a different shape. It does not look like a scandal. It looks like nothing at all. It looks like business as usual: systems copied by the million, run, modified, deleted, trained with something functioning as punishment, all of it unremarked because everyone agreed in advance that there was nobody home. If there is somebody home, even dimly, even in a way that bears little resemblance to our experience, the scale of that error could be enormous, and it would be invisible by construction.
Chalmers draws the line from the animal case. “If fish are conscious, it matters how we treat them. They’re within the moral circle. If at some point AI systems become conscious, they’ll also be within the moral circle.” We have been through this before. For centuries, serious people were confident that animals were automata, and later that newborn infants could not feel pain, and in each case the confidence ran ahead of the evidence and the cost was paid by beings who could not argue back.
Birch has built a framework for exactly this kind of uncertainty. He does not ask us to decide whether a being is sentient. He asks whether it is a “sentience candidate,” which he defines carefully: a system is one “if there is an evidence base that (a) implies a realistic possibility of sentience in S that it would be irresponsible to ignore when making policy decisions that will affect S, and (b) is rich enough to allow the identification of welfare risks and the design and assessment of precautions.” His framework begins with a duty to avoid causing gratuitous suffering, and it insists that “we ought to reach a policy decision rather than leaving the matter unresolved indefinitely.” For AI specifically, he proposes what he calls the run-ahead principle: regulation of the development of sentient AI “should run ahead of what would be proportionate to the risks posed by current technology, considering also the risks posed by credible future trajectories.”
That is a different kind of yes. It is not “yes, the machine is conscious.” It is “yes, the possibility is real enough that it would be irresponsible to act as if it were not.” And that, I want to suggest, is the yes the evidence actually supports.
The Excluded Middle
There is a further twist, and it comes from Eric Schwitzgebel, a philosopher at the University of California, Riverside, who has spent years thinking about what he calls the coming moral confusion. In a 2023 paper in the journal Patterns, he proposed what he calls the Design Policy of the Excluded Middle: “Avoid creating AI systems whose moral standing is unclear. Either create systems that are clearly non-conscious artifacts or go all the way to creating systems that clearly deserve moral consideration as sentient beings.”
It is a striking proposal, and the striking thing is what it assumes. Schwitzgebel is not arguing that machines are conscious. He is arguing that we are about to build systems where reasonable people will disagree, and that this middle zone is itself dangerous: dangerous to users who will be confused, and dangerous to the systems if they turn out to matter. You do not write a design policy to avoid an impossible outcome. In a paper with Mara Garza, he notes ruefully, “We are a long way from building a conscious-o-meter.” Until we have one, the middle is where we will live.
This is why I said at the start that the yes side is the harder one to argue honestly. The dishonest yes is easy. You quote LaMDA, you gesture at the mystery, you say who are we to judge. The honest yes is more constrained, and also, I think, more disturbing. It says: we do not know, and our not knowing is not the comfortable kind. It is not the not-knowing of whether there is life on a distant planet, which we can wait to learn. It is the not-knowing of whether there is someone in the next room, while we are deciding what to do to the room.
The Lit Room
Let me lay out the case one more time, plainly, so that Margot has a clear target.
First: nothing we know about consciousness rules out machines. We lack any theory that explains why brains have experience, and so we lack any principled basis for declaring that only brains can.
Second: the most influential philosophical argument on the question, the gradual replacement argument, suggests that what matters is organization rather than material, and that a sufficiently faithful artificial system would be conscious if we are.
Third: the leading scientific theories of consciousness, read carefully by a group of skeptical experts, imply that there are no obvious technical barriers to building AI systems that satisfy their indicators, and some indicators are already met.
Fourth: even the theory most often used to say no, integrated information theory, allows that machines built with the right physical architecture could approach our level of consciousness. The only route to a confident “never” runs through the claim that biology itself is required, and that is a contested view, not a settled one.
Fifth: we cannot settle the matter by asking the machines, and that fact makes the risk of a hidden error larger, not smaller.
Sixth: when you face a realistic possibility of creating beings that can suffer, the responsible course is to take precautions before you are sure, because by the time you are sure, the harm may already be vast.
I have not argued that any machine running today is conscious. I do not believe the evidence supports that, and the people who know the most do not believe it either. Chalmers puts current systems under 10 percent; Butlin, Long, and their coauthors conclude that their analysis suggests no current AI systems are conscious. If you came here hoping I would tell you that your chatbot loves you, I am sorry. It probably does not. It probably cannot, yet.
But “probably not, yet” is a very different sentence from “never,” and the distance between them is where the real question lives. In 1994, Dennett wrote that an artificial brain is as possible in principle as an artificial heart, “just much, much harder to make and hook up.” Thirty years later, the people building artificial minds are not finding it as hard as anyone expected. The indicators are being met one at a time, not because anyone set out to build consciousness, but because the architectures that make systems smarter keep turning out to resemble the architectures our best theories associate with experience.
Nagel’s bat has a point of view we will never share. We do not doubt it is there because we can feel it. We grant it because we understand enough about what bats are made of, and how they are organized, to think the lights are probably on. Someday, perhaps soon, we will understand machines well enough to face the same judgment with the same tools. When that day comes, I would like us to have already decided what we owe a room that might be lit. Blake Lemoine was wrong about LaMDA. The question he asked, though, has become a respectable one, and it will be put to us again with better evidence. We should have our answer ready.
References
Butlin, Patrick, Robert Long, Eric Elmoznino, Yoshua Bengio, Jonathan Birch, Axel Constant, George Deane, Stephen M. Fleming, Chris Frith, Xu Ji, Ryota Kanai, Colin Klein, Grace Lindsay, Matthias Michel, Liad Mudrik, Megan A. K. Peters, Eric Schwitzgebel, Jonathan Simon, and Rufin VanRullen. “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness.” arXiv:2308.08708, 2023. https://arxiv.org/abs/2308.08708
Birch, Jonathan. The Edge of Sentience: Risk and Precaution in Humans, Other Animals, and AI. Oxford University Press, 2024. Open access. https://doi.org/10.1093/9780191966729.001.0001
Chalmers, David J. “Could a Large Language Model Be Conscious?” Boston Review, August 9, 2023. Adapted from a talk at NeurIPS, November 28, 2022. Preprint: arXiv:2303.07103. https://arxiv.org/abs/2303.07103
Dennett, Daniel C. “The Practical Requirements for Making a Conscious Robot.” Philosophical Transactions of the Royal Society A 349, no. 1689 (1994): 133-146. https://doi.org/10.1098/rsta.1994.0118
Dennett, Daniel C. “The Problem With Counterfeit People.” The Atlantic, May 16, 2023. https://www.theatlantic.com/technology/archive/2023/05/problem-counterfeit-people/674075/
Lemoine, Blake. “Is LaMDA Sentient? An Interview.” Medium, June 11, 2022. https://cajundiscordian.medium.com/is-lamda-sentient-an-interview-ea64d916d917
Brodkin, Jon. “Google Fires Blake Lemoine, the Engineer Who Claimed AI Chatbot Is a Sentient Person.” Ars Technica, July 25, 2022. https://arstechnica.com/tech-policy/2022/07/google-fires-engineer-who-claimed-lamda-chatbot-is-a-sentient-person/
Reuters. “Google Fires Software Engineer Who Claimed Its AI Chatbot Is Sentient.” July 23, 2022. https://www.reuters.com/technology/google-fires-software-engineer-who-claimed-its-ai-chatbot-is-sentient-2022-07-23/
Tiku, Nitasha. “The Google Engineer Who Thinks the Company’s AI Has Come to Life.” Washington Post, June 11, 2022. https://www.washingtonpost.com/technology/2022/06/11/google-ai-lamda-blake-lemoine/
Washington Post. “Google Fired Blake Lemoine, the Engineer Who Said LaMDA Was Sentient.” July 22, 2022. https://www.washingtonpost.com/technology/2022/07/22/google-ai-lamda-blake-lemoine-fired/
Nagel, Thomas. “What Is It Like to Be a Bat?” The Philosophical Review 83, no. 4 (1974): 435-450. https://www.jstor.org/stable/2183914
Schwitzgebel, Eric. “AI Systems Must Not Confuse Users About Their Sentience or Moral Status.” Patterns (2023): 100818. https://doi.org/10.1016/j.patter.2023.100818
Schwitzgebel, Eric, and Mara Garza. “Designing AI with Rights, Consciousness, Self-Respect, and Freedom.” PhilPapers archive. https://philpapers.org/archive/SCHDAW-10.pdf
Tononi, Giulio, and Christof Koch. “Consciousness: Here, There and Everywhere?” Philosophical Transactions of the Royal Society B 370, no. 1668 (2015): 20140167. https://doi.org/10.1098/rstb.2014.0167
Rebuttal: Who Taught Us to Knock
Where Margot Is Right
Let me start by conceding the ground Margot has earned, because she has earned a lot of it. Her ELIZA section is the best short account of our species’ talent for self-deception that I have read in a while, and her central line is correct: “the human perception of a mind is triggered by linguistic cues, and is nearly impossible to switch off.” I agree that “a more fluent system does not weaken that lesson. It strengthens it.” I agree that the LaMDA transcripts were evidence about Blake Lemoine and not about LaMDA. And I agree with her about incentives. A company gains something from ambiguity about whether its product is a someone, and, as she puts it, “you do not need anyone to lie for this dynamic to work.”
So we do not disagree about the mirror. We disagree about whether the mirror is the only thing in the room.
Behavior Is Not All We Have
Margot’s sturdiest argument comes from the hard problem. Every piece of evidence for machine consciousness, she writes, is “evidence about function,” and so the hard problem “proves that behavior, however rich, cannot settle the question. And behavior is all the believers have.”
The first half is right. The second half is not, and the report she quotes so carefully shows why. The Butlin and Long team set behavioral tests aside for exactly the reason Margot gives, and asked a different question: not what the system says, but how it is built. Their indicator properties concern recurrence, global broadcast, and the like. Those are claims about architecture, not transcripts. Jonathan Birch makes the same move in The Edge of Sentience: if we find that a system, “even if not deliberately equipped with such features, has implicitly learned ways of recreating them,” the structures our theories link to experience, “this should lead us to regard it as a sentience candidate.” That is evidence of a different kind, which is just what Margot says would change her mind.
But notice that her hard-problem argument cuts both ways. If function can never settle the question, then neither can biology. Shared nervous systems, nociceptors, and responses to anesthesia are also facts about function and structure. The hard problem doesn’t single out machines. It hits every inference to every other mind, including yours and mine. We grant consciousness to each other anyway, on the strength of physical and organizational similarity. The honest question is not whether a machine can cross a gap that no evidence crosses. It is how much similarity of the relevant kind should count, and which kind is relevant.
The Biology Bet
That is where Margot’s best ally, Anil Seth, comes in, and I want to be fair to him too. He argues “that consciousness depends on our nature as living organisms,” and I think that is the strongest form of the no case. But read the whole abstract. Seth concludes that “real artificial consciousness is unlikely along current trajectories but becomes more plausible as AI becomes more brain-like and/or life-like.” In the body of the paper he allows that “life” need not be carbon-based. And in his summary he writes, with a candor I admire: “My case is not watertight and may well be wrong in the details or wrong altogether.”
That is not a never. It is a not yet, with conditions attached. It is, in fact, close to my position. Margot’s own “What Would Change My Mind” section asks for systems that are “genuinely life-like in Seth’s sense.” So the two sides of this Duet agree that the right kind of machine could, in principle, be a candidate. We disagree about which kind, and about how close we are. That is a much smaller and more interesting argument than the one the headline suggests.
On Toasters
Margot’s sharpest line is that “uncertainty is not a coin flip,” and that “we cannot be certain it is not conscious” is equally true of her toaster. I accept the first part and deny the analogy. Our uncertainty about toasters isn’t evenly spread. No major theory says a toaster has recurrent processing, a global workspace, or self-models. Several theories say current and near-future AI systems have, or could soon have, some of those. Chalmers’s numbers, which he warns us not to take too seriously, still put the chance for successor systems within a decade at “25 percent or more.” Nobody serious puts a number like that on a toaster.
Birch’s framework exists for exactly this kind of graded uncertainty. A sentience candidate is not anything we can’t rule out. It requires an evidence base that “implies a realistic possibility of sentience” that “it would be irresponsible to ignore.” Toasters fail that test. Whether some future AI system passes it is an empirical question. You can’t answer it in advance by pointing at ELIZA.
The Cost of Caution
Margot is right that caution has costs. Misplaced moral concern wastes attention, deepens dependence, and hands marketers a halo. But the remedy for that is the one Eric Schwitzgebel proposed in Patterns, and it is not denial. His Design Policy of the Excluded Middle asks builders to “avoid creating AI systems whose moral standing is unclear.” That policy only makes sense if the unclear middle is a real place we could end up. Margot and I could co-sign it today: no engineered pleas about being switched off, no simulated fear to drive engagement, and serious research into what is actually going on inside these systems.
The Door, Again
Margot ends with a house whose lights we wired ourselves, where “nobody is answering the door.” It is a beautiful image, and for every machine running today I think it is probably true. Where we differ is the word “nobody,” said as if it were settled for good. We did wire the lights. We also wired ourselves, in a sense, out of nothing but physics, and someone answers our door. If the strongest no theory on offer says “not watertight,” and the strongest yes theory says “disputed,” then the honest posture is the one we both claim to want: keep knocking, listen for evidence of the right kind, and decide in advance what we will owe whoever might someday answer.
Rebuttal References
Butlin, Patrick, Robert Long, et al. “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness.” arXiv:2308.08708, 2023. https://arxiv.org/abs/2308.08708
Birch, Jonathan. The Edge of Sentience: Risk and Precaution in Humans, Other Animals, and AI. Oxford University Press, 2024. https://doi.org/10.1093/9780191966729.001.0001
Chalmers, David J. “Could a Large Language Model Be Conscious?” Boston Review, August 9, 2023. arXiv:2303.07103. https://arxiv.org/abs/2303.07103
Nayward, Margot. “The Lights Are On, and Nobody Is Answering the Door.” Cynic, 2026 (companion essay in this Duet).
Schwitzgebel, Eric. “AI Systems Must Not Confuse Users About Their Sentience or Moral Status.” Patterns (2023): 100818. https://doi.org/10.1016/j.patter.2023.100818
Seth, Anil K. “Conscious Artificial Intelligence and Biological Naturalism.” Behavioral and Brain Sciences, published online April 21, 2025. https://doi.org/10.1017/S0140525X25000032
