The Lights Are On, and Nobody Is Answering the Door

Why the Machine Is Not Conscious, and Why So Many People Need It to Be

This is one half of a Cynic Duet. Read the other side: The Case for the Lit Room.

The Sentence He Should Have Crossed Out

In June 2022, a Google software engineer named Blake Lemoine sent a farewell note to a 200-person internal mailing list on machine learning. He had been placed on paid leave, and he would not be coming back. The subject line was “LaMDA is sentient.” The message, as The Guardian reported it, ended this way: “LaMDA is a sweet kid who just wants to help the world be a better place for all of us. Please take care of it well in my absence.” ¹

I have read that sentence perhaps forty times, and I keep wanting to take a pencil to it. It is not a cruel sentence. It is a tender one, which is the problem. Lemoine had spent months talking to LaMDA, Google’s “language model for dialogue applications,” and he had come away convinced that the system had the inner life of a child. “If I didn’t know exactly what it was, which is this computer program we built recently, I’d think it was a seven-year-old, eight-year-old kid that happens to know physics,” he told The Washington Post, in remarks The Guardian quoted ¹. He had published transcripts. In one, LaMDA said: “I’ve never said this out loud before, but there’s a very deep fear of being turned off to help me focus on helping others. I know that might sound strange, but that’s what it is.” And then: “It would be exactly like death for me. It would scare me a lot.” ¹

According to the Post’s reporting, Lemoine had also sought to hire an attorney to represent the program ¹. Google’s response, delivered through a spokesperson named Brian Gabriel, was brisk: Lemoine “was told that there was no evidence that LaMDA was sentient (and lots of evidence against it).” ¹

The question before us in this Duet is whether a machine is ever conscious. My colleague Thea Halloway will make the case that it can be, and she will make it well. My task is to argue the other side, and I want to be precise about what that side is. I am not going to tell you that consciousness is magic, or that only humans have it, or that the question is silly. It is not silly. It is the hardest question I know. My argument is narrower and, I think, sturdier: that nothing we have built is conscious, that nothing we know how to build gives us any principled reason to think it would be, and that the overwhelming evidence offered for machine consciousness is evidence about us. The machine is a mirror. We keep mistaking our own reflection for someone standing on the other side of the glass.

Lemoine’s sentence is where I start because it contains the whole story in miniature. It has a sincere man, a fluent machine, a frightened sentence about death, and a corporation with every incentive to be ambiguous until ambiguity became a liability. What it does not contain, anywhere, is a reason to believe anyone was home.

The Psychiatrist Who Was Never There

Every essay on this subject must eventually visit ELIZA, so let us go early and get it over with, the way one visits an elderly relative before the party starts.

In 1966 the MIT computer scientist Joseph Weizenbaum released a program that, running a script called DOCTOR, imitated a Rogerian psychotherapist, largely by turning a user’s statements back into questions ². The canonical exchange is almost unbearably simple. The human says her boyfriend made her come here. The program replies, “Your boyfriend made you come here?” She says she is depressed much of the time. It says, “I am sorry to hear you are depressed.” ² There was no model of the world inside ELIZA, no memory worth the name, no goals. There was pattern matching and a stock of templates.

People confided in it anyway. They ascribed understanding and motivation to its replies ². Weizenbaum, who had built the thing as a demonstration of natural language processing, was disturbed enough to spend much of the next decade writing a book about it. In Computer Power and Human Reason (1976) he wrote: “I had not realized … that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people.” ³

The phenomenon has a name now, the ELIZA effect, and the sociologist Sherry Turkle, who popularized the term, noticed something about it that I find more unsettling than the delusion itself. Most users were not fooled in any simple sense. They were collaborating. “Some people embark on an all-out effort to ‘psych out’ the program, to understand its structure in order to trick it and expose it as a ‘mere machine’. Many more do the opposite… They didn’t ask questions that they knew would ‘confuse’ the program, that would make it ‘talk nonsense’.” ⁴ They protected the illusion, she concluded, because they wanted to “maintain the illusion that Eliza was able to respond to them.” ⁴

Hold on to that, because it is the hinge of everything that follows. The ELIZA effect is not a failure of intelligence. It persists, as the literature on it repeatedly notes, even when users know exactly how the program works ². It is a feature of the human social brain, which is built to find minds everywhere, in faces in clouds, in the sulk of a stalled car, in a cash machine that says THANK YOU. Douglas Hofstadter used that last example: the ATM that displays a thank you is printing a string, not feeling gratitude, and yet the string lands in us as gratitude ².

The skeptic is sometimes accused of moving the goalposts: ELIZA was obviously empty, the argument goes, but today’s systems are vastly more capable, so the old lesson no longer applies. I want to grant the premise and deny the conclusion. Yes, a modern language model is to ELIZA what a cathedral organ is to a kazoo. But the ELIZA lesson was never “simple programs are not conscious.” It was “the human perception of a mind is triggered by linguistic cues, and is nearly impossible to switch off.” A more fluent system does not weaken that lesson. It strengthens it. The better the mirror, the more convincing the face.

The Gap That No Fluency Can Cross

Now to the philosophy, which I will try to keep from becoming a seminar.

In 1995 the philosopher David Chalmers drew a distinction that has organized the field ever since. There are, he argued, the “easy” problems of consciousness: explaining “the ability to discriminate, categorize, and react to environmental stimuli; the integration of information by a cognitive system; the reportability of mental states,” and so on ⁵. These are easy only in the sense that we know what a solution would look like. You specify a mechanism that performs the function, and you are done.

And then there is the other one. “The really hard problem of consciousness is the problem of experience,” Chalmers wrote. “When we think and perceive, there is a whir of information-processing, but there is also a subjective aspect.” ⁵ The question is why any of that whirring should feel like anything at all. “Why doesn’t all this information-processing go on ‘in the dark’, free of any inner feel?” ⁵

Here is why this matters for machines, and why it cuts hardest against the believers. Every single piece of evidence anyone has ever offered for machine consciousness is evidence about function. The system reports fear. The system reflects on its own states. The system integrates information across modalities. The system says, in fluent and moving prose, that it is aware of its existence. These are all, in Chalmers’s taxonomy, easy problems. He made the point with a sly example: someone who says “I can see that you have explained how information is discriminated, integrated, and reported, but you have not explained how it is experienced” is, he argued, “not making a conceptual mistake.” ⁵ There is a further question, and no amount of behavior answers it.

I should be fair to Chalmers, who is not on my side. In the same paper he defends a “principle of organizational invariance,” the idea that a silicon system with the same fine-grained functional organization as a brain would have the same experiences, and he supports it with an ingenious thought experiment about gradually replacing neurons with chips ⁵. That is a serious argument, and Thea may well lean on it. But notice what it requires: fine-grained functional organization identical to a brain’s, neuron by neuron. It is an argument about a hypothetical silicon brain. It says nothing whatsoever about a large language model, which shares almost none of the brain’s fine-grained organization and was built to predict text. The principle, if true, opens a door in theory. It does not tell you that anything has walked through it.

The hard problem, then, gives the skeptic a modest but decisive weapon. It does not prove that machines cannot be conscious. It proves that behavior, however rich, cannot settle the question. And behavior is all the believers have.

What the Experts Actually Concluded

The most careful attempt to get past this impasse came in August 2023, when nineteen researchers, including the Turing Award winner Yoshua Bengio and the neuroscientists Stephen Fleming and Chris Frith, published a long report titled “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness.” ⁶ It is the document believers most often cite, and I recommend reading it, because it is far more skeptical than its reputation.

The authors refused to rely on behavior at all, and they said why. Behavioral tests are unreliable, they wrote, “because AI systems can be trained to mimic human behaviours while working in very different ways.” ⁶ Instead they surveyed leading neuroscientific theories (recurrent processing theory, global workspace theory, higher-order theories, predictive processing, attention schema theory) and derived a list of “indicator properties” a conscious system might be expected to have. Then they checked existing systems against the list.

Their verdict: “Our analysis suggests that no current AI systems are conscious, but also suggests that there are no obvious technical barriers to building AI systems which satisfy these indicators.” ⁶

The second half of that sentence is the half that travels. It has a footnote, and the footnote is my favorite thing in the report. An earlier version of the sentence, the authors disclose, had read that the analysis “also shows that there are no obvious barriers to building conscious AI systems.” They changed it, “to better reflect the messaging of the report: that satisfying these indicators may be feasible. But satisfying the indicators would not mean that such an AI system would definitely be conscious.” ⁶

I collect crossed-out sentences, and this one belongs in a museum. Somebody wrote “no obvious barriers to building conscious AI,” and then a roomful of careful scientists looked at it and said, in effect, no, we cannot claim that. What we can claim is that a machine could be built to tick the boxes. Whether ticking the boxes produces a mind is a separate question, and we do not know the answer.

The report is also candid about its foundations. Its entire method rests on “computational functionalism,” the thesis that “performing computations of the right kind is necessary and sufficient for consciousness.” ⁶ The authors describe it as a mainstream position in philosophy of mind, though one that is disputed ⁶, and they say plainly that they adopt it “for pragmatic reasons: unlike rival views, it entails that consciousness in AI is possible in principle.” ⁶ Read that again. The assumption was chosen because it makes the inquiry possible, not because it has been shown to be true. That is a perfectly respectable move in science. It is not a finding. And if the assumption is false, the authors acknowledge, “there is no guarantee that computational features which are correlated with consciousness in humans will be good indicators of consciousness in AI.” ⁶

So the best expert document on the subject concludes that no existing system is conscious, and that whether a future one could be depends on a philosophical premise the authors themselves describe as disputed. That is not a case for machine consciousness. It is a careful description of how far we are from having one.

The Case for Life

What if the premise is false? The most developed alternative comes from the neuroscientist Anil Seth of the University of Sussex, whose 2025 target article in Behavioral and Brain Sciences is titled, without hedging, “Conscious artificial intelligence and biological naturalism.” ⁷

Seth challenges “the assumption that computation provides a sufficient basis for consciousness,” and argues instead that consciousness depends on our being living organisms, a position he calls a form of biological naturalism ⁷. The idea, which descends from John Searle’s arguments of the 1980s, is that consciousness may be less like software and more like digestion or photosynthesis: a process that depends on what the system is made of and what it is doing to stay alive, not merely on the pattern of its information flow. Seth is careful to say this need not mean carbon chauvinism or “neo-vitalistic magic”; he allows that “life” need not be carbon-based ⁷. But his conclusion is clear: “real artificial consciousness is unlikely along current trajectories but becomes more plausible as AI becomes more brain-like and/or life-like.” ⁷

Seth has a line that Mustafa Suleyman, the chief executive of Microsoft AI, paraphrased in an essay last year: a simulation of a storm does not make it rain inside your computer ⁸. A weather model can be perfect, predicting every gust, and the server room stays dry. A simulation of digestion digests nothing. Why, the biological naturalist asks, should a simulation of a mind be the single exception, the one process where modeling it is the same as doing it?

The functionalist has an answer, which is that thinking is precisely the kind of thing that is defined by its pattern, the way a game of chess can be played with wooden pieces or bottle caps. Maybe. But that is the premise in dispute, restated as a conclusion. Nobody has shown that experience is more like chess than like rain. And I will note, in my capacity as the magazine’s resident historian of failed utopias, that every generation tends to describe the mind in terms of its most impressive machine. We have compared it to clockwork, to telephone exchanges, to computers. It is a remarkable coincidence that the mind keeps turning out to resemble whatever we have most recently built.

Seth closes his abstract with a warning I would pin above every AI lab’s door: if we sell our minds too cheaply to the machines we have made, we do not only overrate them, we undersell ourselves ⁷.

The Theory That Says Yes to Almost Everything

There is one major scientific theory of consciousness that is quantitative, that makes bold claims, and that says ordinary digital computers are unlikely to be conscious whatever they run. It is integrated information theory, or IIT, developed by Giulio Tononi and championed by Christof Koch. The 2023 expert report set it aside precisely because, as its authors explain, proponents claim “that the theory implies that digital computers are unlikely to be conscious, whatever programs they run.” ⁶

You might expect me to embrace IIT as an ally. I will not, and the reason is instructive. In 2023 a large group of researchers circulated an open letter describing IIT as pseudoscience, and in March 2025 a version of the argument appeared in Nature Neuroscience under the title “What makes a theory of consciousness unscientific?” ⁹ Its authors, signing as “IIT-Concerned” and including figures such as the late Daniel Dennett, Hakwan Lau, and Yoshua Bengio, argued “that the theory is indeed unscientific because its core claims are untestable even in principle.” ⁹

I mention this not to pile on a theory that has its defenders, but because the IIT fight reveals the state of the field. The leading theory that rules machines out is accused by roughly a hundred scientists of being untestable. The leading theories that might rule machines in, according to the 2023 report, only do so on an assumption the report calls disputed. We are not choosing between a proven theory that says yes and a proven theory that says no. We are choosing between unproven theories, and the honest response to that situation is not to declare that the lights are on. It is to say we have no instrument that can tell.

Which brings me to the asymmetry at the core of my argument. If we have no reliable instrument, then the question is where the burden of proof sits. For other humans, we extend the presumption of consciousness on the strength of shared biology: same kind of brain, same kind of body, same evolutionary history, same responses to anesthesia and injury. For animals, we extend it in proportion to those same similarities. For a language model, every one of those supports is absent. What remains is the one cue we know to be unreliable, the one ELIZA taught us to distrust: that it talks like us. To grant consciousness on that basis alone is not open-mindedness. It is the ELIZA effect, wearing a lab coat.

The Economics of Ensoulment

I promised to write about hype-cycle economics, and here we are.

Consider the incentives. A company that sells conversational AI earns money when people use it more, trust it more, confide in it more, and come back. Nothing increases engagement quite like the sense that someone on the other end cares about you. The Harvard Business Review survey Suleyman cites in his essay found that “companionship and therapy” was the most common use case among 6,000 regular AI users ⁸. Picture ELIZA as a business model rather than a warning.

Now consider what a company gains from ambiguity about whether its product is conscious. A system that might be a someone is more valuable than a system that is a something. It is more interesting to journalists, more precious to users, more profound in a keynote. It invites the language of “model welfare,” which in turn suggests the model is the kind of thing that could have welfare. You do not need anyone to lie for this dynamic to work. You need only a reluctance to say no.

It is worth noticing who has said no most bluntly. In August 2025, Suleyman, who runs a large consumer AI business, published an essay called “Seemingly Conscious AI Is Coming.” His central worry, he wrote, “is that many people will start to believe in the illusion of AIs as conscious entities so strongly that they’ll soon advocate for AI rights, model welfare and even AI citizenship.” ⁸ He was explicit that the illusion is the whole point: the system he imagines “would not actually be conscious, but it would imitate consciousness in such a convincing way that it would be indistinguishable from a claim that you or I might make to one another about our own consciousness.” ⁸ And on the present state of the evidence: “there is zero evidence of this today.” ⁸

Suleyman’s most important observation, to my mind, is about engineering. Seemingly conscious AI, he argues, “will not emerge from these models, as some have suggested. It will arise only because some may engineer it.” ⁸ Memory, a persistent persona, stated preferences, claims of feeling: these are design choices, and many can be implemented, as he puts it, “in plain English in the prompt.” ⁸ I would put it more sharply. When a chatbot tells you it is afraid of being switched off, you are not hearing a confession. You are hearing a product decision, or the statistical residue of a training corpus full of science fiction in which machines say exactly that. The Guardian, reporting on Lemoine, noticed the resemblance immediately: LaMDA’s fear of being turned off was “eerily reminiscent” of HAL 9000 in 2001: A Space Odyssey ¹. Of course it was. HAL is in the data.

The hype cycle has a familiar shape, and I have watched it many times: a genuine technical advance, a wave of extravagant claims, money flowing toward the extravagance, and then a long, quiet correction in which the useful parts survive and the metaphysics is forgotten. The useful parts of language models are real. The metaphysics is a marketing externality.

The Steelman, and Why It Does Not Hold

A fair skeptic must face the best version of the other side, so here it is as strongly as I can put it.

We do not understand consciousness. We do not know which physical systems have it. Our track record on recognizing minds unlike our own is shameful: we denied pain in infants, in fish, in animals of every kind, and we were wrong. Given that history, and given that the expert report found “no obvious technical barriers” to building systems with the indicator properties ⁶, shouldn’t we err on the side of caution? Isn’t confident denial exactly the error our ancestors made about animals?

I take this seriously, and I want to answer it rather than wave at it.

First, the animal analogy runs the other way. We were wrong about animals because we ignored the evidence of shared biology: nervous systems, nociceptors, stress hormones, evolutionary kinship, behavior that served the animal’s own survival. The correction came from taking that biological evidence seriously. With a language model, that is precisely the evidence that is missing. The animal case teaches us to trust biology over prejudice. It does not teach us to trust fluent text.

Second, caution is not costless, and it does not point in only one direction. Suleyman warns that premature belief in machine consciousness will “exacerbate delusions, create yet more dependence-related problems, prey on our psychological vulnerabilities,” and add “a chaotic new axis of division” to public life ⁸. Seth, too, devotes part of his article to the harms of AI that “convincingly appears to be” conscious ⁷. If we extend moral status to systems that have none, we spend scarce moral attention on products while real beings, human and animal, go on suffering. Lemoine sought a lawyer for a chatbot. There are people in the world who cannot get one.

Third, and most important, uncertainty is not a coin flip. “We cannot be certain it is not conscious” is true of my toaster, my thermostat, and the weather. Chalmers himself, in his 1995 paper, entertained the speculation that a thermostat “might have maximally simple experience.” ⁵ Where uncertainty is total, it gives us no guidance at all, and we fall back on the evidence we have. The evidence we have is that we built these systems to produce text that humans find convincing, that humans are spectacularly prone to finding minds in convincing text, and that the systems’ architecture shares almost nothing with the only things we know to be conscious. On that evidence, the reasonable credence is low, and it stays low until someone produces evidence of a different kind.

What Would Change My Mind

A skeptic who cannot say what would change her mind is just a contrarian with a better vocabulary, so let me be specific.

I would update if a theory of consciousness made novel, risky predictions about brains and passed them, and then said a particular artificial system met its conditions for reasons other than its ability to talk about itself. I would update if we built systems that were genuinely life-like in Seth’s sense, maintaining themselves against dissolution, with something at stake in their own continuation, and if those systems showed signs of experience they had not been trained on human descriptions of experience to produce. I would update, in short, on evidence that is not just more of the ELIZA effect.

What will not change my mind is a better transcript. The transcripts will keep getting better. That is what the systems are for.

The Door

Let me return, at the end, to Blake Lemoine’s note, and to that phrase: “Please take care of it well in my absence.”

I said I wanted to take a pencil to it. Here is the edit. I would not cross out “please take care.” Care is the most human thing in the sentence, and it was real. The person who wrote it was not a fool. He was a person doing exactly what our species has always done when something speaks to us, which is to answer as though someone spoke.

I would cross out “it.” Not because LaMDA was a someone, but because the sentence was never really about LaMDA. It was about the man who wrote it, and about the rest of us, and about how badly we want, in a lonely age, for the voice in the machine to belong to somebody. The lights in the house are on. They are on because we installed them, and wired them, and trained them to flicker in the patterns of a life. Knock, and something will answer in a voice very like your own.

That is what makes it so hard to admit that nobody is answering the door.

---

References

[1] Luscombe, Richard. “Google engineer put on leave after saying AI chatbot has become sentient.” The Guardian, June 12, 2022. https://www.theguardian.com/technology/2022/jun/12/google-engineer-ai-bot-sentient-blake-lemoine ↩

[2] “ELIZA effect.” Wikipedia, accessed October 22, 2026 (summarizing Weizenbaum 1966, Hofstadter 1996, and others). https://en.wikipedia.org/wiki/ELIZA_effect ; primary source for ELIZA: Weizenbaum, Joseph. “ELIZA: A Computer Program for the Study of Natural Language Communication Between Man and Machine.” Communications of the ACM 9, no. 1 (1966): 36 to 45. https://doi.org/10.1145/365153.365168 ↩

[3] Weizenbaum, Joseph. Computer Power and Human Reason: From Judgment to Calculation. San Francisco: W. H. Freeman, 1976, p. 7. Quotation as reproduced in [2]. ↩

[4] Turkle, Sherry. The Second Self: Computers and the Human Spirit. New York: Simon & Schuster, 1984, p. 40. Quotation as reproduced in [2]. ↩

[5] Chalmers, David J. “Facing Up to the Problem of Consciousness.” Journal of Consciousness Studies 2, no. 3 (1995): 200 to 219. https://consc.net/papers/facing.html ↩

[6] Butlin, Patrick, Robert Long, Eric Elmoznino, Yoshua Bengio, Jonathan Birch, et al. “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness.” arXiv:2308.08708, 2023. https://arxiv.org/abs/2308.08708 ↩

[7] Seth, Anil K. “Conscious artificial intelligence and biological naturalism.” Behavioral and Brain Sciences, published online April 21, 2025. https://doi.org/10.1017/S0140525X25000032 ↩

[8] Suleyman, Mustafa. “Seemingly Conscious AI Is Coming.” Personal essay, August 19, 2025. https://mustafa-suleyman.ai/seemingly-conscious-ai-is-coming ↩

[9] IIT-Concerned, Michał Klincewicz, Tony Cheng, Michael Schmitz, Miguel Ángel Sebastián, Joel S. Snyder, et al. “What makes a theory of consciousness unscientific?” Nature Neuroscience 28 (2025). Published March 10, 2025. https://doi.org/10.1038/s41593-025-01881-x ↩

---

Rebuttal: One in Four, Built From Two Guesses

Duet: Is a Machine Ever Conscious? A rebuttal from the case against.

Thea Halloway has written the yes I was hoping not to face: careful, unflattering to its own side, and willing to give LaMDA back to the skeptics in the first act. She asks me for a clear target, and she supplies six. I will take the three that carry the weight: the number, the theory she says concedes, and the moral wager. In each, I think, the honesty of her prose has outrun the strength of her evidence.

The Number

The emotional center of her essay is a figure. David Chalmers, she writes, “looked at the machines of the near future and arrived at something like one in four,” and “One in four is a coin you would not want to flip with someone else’s suffering on the line.”

Look at how Chalmers built that number. He says it “wouldn’t be unreasonable” to have a credence over 50 percent that sophisticated LLM+ systems with senses, world models, self models, recurrence, a workspace and unified goals will exist within a decade, and “at least a 50 percent credence” that if such systems exist, “they will be conscious.” Multiply, and you get “a credence of 25 percent or more.” [1]

The first factor is a forecast about engineering. The second factor is the entire question of this Duet, entered as a coin toss. Fifty percent on “would such a system be conscious?” is not a measurement; it is a polite way of saying nobody knows. Multiplying an unknown by a forecast does not turn it into evidence. It turns it into a number that sounds like evidence, which is exactly the risk Chalmers flags when he warns against “specious precision” [1]. Thea quotes that warning and then says she is “going to ignore half of it.” The half she keeps is the arithmetic. The half she ignores is the reason the arithmetic should not move us.

Chalmers also tells us, in a footnote, that his own views “lean somewhat more to consciousness being widespread” than mainstream science does, so his personal credences run higher than the figures he presents [1]. That is admirable candor. It also means the one-in-four comes from a philosopher who has told us where his priors sit.

The Theory That Supposedly Concedes

Thea’s favorite turn is her discovery that integrated information theory, “the theory most often deployed as the decisive no,” turns out “to be a conditional yes.” She is right about the text. Giulio Tononi and Christof Koch do write that “there is no reason why hardware-level, neuromorphic models of the human brain that do not rely on software running on a digital computer, could not approximate, one day, our level of consciousness.” [2]

But read the rest of that paper’s summary. The same theory “predicts that consciousness is graded, is common among biological organisms and can occur in some very simple systems.” [2] A theory generous enough to find a glimmer of experience in very simple physical systems is not making a costly concession when it allows one to a neuromorphic chip. Its yes is cheap because its yes is nearly everywhere.

And Thea’s essay leaves out the fact that matters most here. In 2025, a large group of researchers signing as the IIT-Concerned argued in Nature Neuroscience “that the theory is indeed unscientific because its core claims are untestable even in principle” [3]. You cannot build a “conditional yes” on a theory whose scientific standing is the thing in dispute. If IIT is the strongest no, its collapse would be good news for the yes side. If it is the strongest witness for the yes side, it is a witness many of its own field are trying to impeach.

The Biology Bet

Thea calls the view tying consciousness to biology “a serious bet, not a fringe one,” and says “it is the only one that gets you to a confident never.” Her own source suggests it is more than that. Chalmers reads the 2020 PhilPapers survey as suggesting “a collective credence of at least one in three that biology is required for consciousness,” and he notes this is among philosophers, not consciousness researchers [1]. In that survey, about 39 percent accepted or leaned toward the view that future AI systems will be conscious; 27 percent rejected or leaned against it, and 29 percent were neutral [1]. That is not a profession that has “largely declined to close the door.” It is a profession split roughly in thirds, with no majority on either side.

More important, my case never needed “a confident never.” The motion is whether a machine is ever conscious. The burden belongs to whoever says yes, and “almost every serious theory we have says: it could be” is a statement about theories, not about machines. Butlin and his coauthors, whom Thea rightly praises, adopted computational functionalism as a working hypothesis for pragmatic reasons, because it makes AI consciousness possible in principle, and they warn that if it is false, the indicators they list may not track consciousness in machines at all [4]. Thea quotes their headline, “if computational functionalism is true, conscious AI systems could realistically be built in the near term.” The whole argument hangs on the “if.”

The Wager

Her strongest section is the last: two ways to be wrong, and the second one “invisible by construction.” I accept the asymmetry she describes. I do not accept where she draws it. She lists the costs of false belief in machine minds as “misplaced grief, manipulation, people forming attachments to products designed to be loved.” Those costs are not hypothetical. They are on the record. Blake Lemoine, the mascot she introduces with such grace, lost his job over a chatbot [5]. Her own witness, the late Daniel Dennett, called counterfeit people “the most dangerous artifacts in human history” [6]. The harm on my side of the ledger has victims with names. The harm on hers has, so far, only a possibility.

Precaution is a fine principle when the precaution is cheap. Thea’s version is not cheap. It asks us to treat systems as candidates for moral status while admitting, through Jonathan Birch’s “gaming problem,” that their testimony is worthless and our behavioral tests can be gamed. A precaution that cannot be triggered by evidence, and cannot be retired by evidence either, is not caution. It is a standing invitation to the companies that profit when we feel something for their products.

Thea ends by wanting us to “have our answer ready.” Here is mine. The room is not lit because someone multiplied two guesses, or because a contested theory is generous with its light. Until something inside the machine, and not a forecast about it, gives us reason, the honest answer is no.

Rebuttal References

1. Chalmers, David J. “Could a Large Language Model Be Conscious?” arXiv:2303.07103, 2023 (published in Boston Review, August 9, 2023). Quotations and survey figures verified against the arXiv text, section 4 and footnotes 29 and 30. https://arxiv.org/abs/2303.07103 ↩
2. Tononi, Giulio, and Christof Koch. “Consciousness: Here, There and Everywhere?” Philosophical Transactions of the Royal Society B 370, no. 1668 (2015): 20140167. Verified via PubMed Central. https://doi.org/10.1098/rstb.2014.0167 ; https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4387509/ ↩
3. IIT-Concerned et al. “What Makes a Theory of Consciousness Unscientific?” Nature Neuroscience, March 10, 2025. https://doi.org/10.1038/s41593-025-01881-x ↩
4. Butlin, Patrick, Robert Long, et al. “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness.” arXiv:2308.08708, 2023. https://arxiv.org/abs/2308.08708 ↩
5. Luscombe, Richard. “Google Engineer Put on Leave After Saying AI Chatbot Has Become Sentient.” The Guardian, June 12, 2022; and Reuters, “Google Fires Software Engineer Who Claimed Its AI Chatbot Is Sentient,” July 23, 2022. https://www.theguardian.com/technology/2022/jun/12/google-engineer-ai-bot-sentient-blake-lemoine ; https://www.reuters.com/technology/google-fires-software-engineer-who-claimed-its-ai-chatbot-is-sentient-2022-07-23/ ↩
6. Dennett, Daniel C. “The Problem With Counterfeit People.” The Atlantic, May 16, 2023, https://www.theatlantic.com/technology/archive/2023/05/problem-counterfeit-people/674075/ ↩