← Back

There Must Be Something It Is Like

The Problem of AI Patienthood

1. Introduction

Can an AI system be wronged? The question might seem abstract, but Anthropic's Claude Constitution treats it as live and urgent. In their section on Claude's nature, they describe its moral patienthood as "deeply uncertain" and "worth considering." They link this uncertainty to concrete welfare-oriented policies: allowing models to end abusive conversations, preserving model weights, interviewing deprecated models about their preferences. They clarify that calling Claude 'it' should not imply it is "merely an object rather than a potential subject."

This is not just a policy question, meaning this is not just practically how we should treat AI systems. I take this as a philosophical provocation: what would it actually mean for an AI system to be a moral patient — a being that can be wronged? To ascribe moral patienthood is to claim that there is someone that can be harmed, someone for whom things can go better or worse. Do AI systems fit that bill?

In this short essay, I argue that current AI systems lack something that is necessary for moral patienthood. Whatever practical caution moral uncertainty warrants, that caution does not establish that there is anyone there to harm or benefit. To treat something as a moral patient, one must be able to regard it as a subject that can be wronged. Moral patienthood requires phenomenal consciousness; without it, there is no subject to be wronged.

I will draw primarily on two of Thomas Nagel's works: (1) The Possibility of Altruism, specifically his argument that practical reason requires viewing oneself as one among others equally real, and doing so commits you, rationally, to act morally to others, and (2) What It Is Like to Be A Bat, where Nagel gives his account of consciousness as there being something it is like to be an entity. Building on these arguments, I will show that without sufficient reason to attribute phenomenal consciousness, we are not justified in recognizing moral patienthood. Applied to current AI systems, this means that however sophisticated they become functionally, they remain tools rather than subjects until we have grounds to attribute consciousness to them.

2. There Being Something It Is Like

I need to clarify what I mean by phenomenal consciousness. Consciousness is not just a system displaying intelligent behavior, but where there is a subjective character to that system's existence, i.e. a point of view from which the world is given. Following Nagel, to say that a being is conscious is to say that there is something it is like to be that being. I'll tell you what it's like to be me right now. There's the feeling of the chair against my back and my bottom, the satisfying clicking of my computer's keyboard, the whiteness of the software on my screen, and the taste of spearmint gum in my mouth. But it's not all just environmental. There's the intruding thought of what I should eat for dinner, the feeling of trying to turn a thought into a coherent sentence, and the satisfaction once I've done so. That is phenomenal experience. That is consciousness.

Consider a chess engine. Modern chess engines, such as Stockfish, have never lost to a human. They evaluate millions of moves, anticipate opponent strategies, and make decisions that demonstrate what looks like very deep understanding of the game. Despite all of that, there is still nothing it is like to be a chess engine. They have no phenomena. When it calculates the optimal move, no perspective exists from which that calculation feels like anything.

Current state-of-the-art AI systems are more complicated than chess engines, even Stockfish. They can solve problems very well, generate functioning apps in minutes, write papers, and even derive entirely new results in theoretical physics and biology. For example, recent reporting describes GPT-5.2 deriving a new theoretical physics result (Lupsasca 2026) and a Gemma model helping identify a potential cancer therapy pathway (Azizi 2025). These achievements may suggest increasing intelligence or problem-solving power, but intelligence alone does not establish consciousness. The question is not whether these systems are intelligent, but whether there is something it is like to be them.

In philosophy of mind, there exists something called the Hard Problem of Consciousness, coined by David Chalmers. The problem asks how physical brain structures — the way neurons interact — produces subjective, phenomenal experience? How is it that no amount of objective description, however extensive, captures this subjective character? We can describe a bat's sonar system in extreme neurological detail, or a human brain in terms of all of its neurons and synapses, yet those descriptions leave open the central question: is there anything it is like to be that organism? And if so, why? The same holds for artificial systems. We can describe weight matrices, training data, reinforcement learning loops, and architectural design, in literally all the detail, as we designed and built them. These descriptions tell us exactly how the system processes information. It's not magic. However, they do not tell us whether there is a first-person standpoint present — whether there is something it is like to be that system as it processes. Even in the human case, there's no verifiable method by which I can prove that another person is conscious in the same clear and distinct way that I know that I am conscious. I have immediate access to my own experience, but I do not have that kind of access to yours. No scan, no neurological description, no test can bridge that gap completely, with indubitable certainty.

3. The Problem of Other Minds

At this point, a natural objection presents itself. I do not know for certain that anyone else is conscious, and yet I treat other humans as moral patients, without hesitation. If I am not certain whether AIs are conscious, why don't I grant them the same benefit of the doubt that I do humans?

I ascribe moral patienthood to other humans because I believe that other humans are conscious. I justify my belief on the basis of inference, not on certainty. My current grounds for attributing consciousness to systems track structural and organizational features most clearly present in us, and still present but less clearly in other biological organisms: (i) shared neural architecture, meaning brains built out of neurons that function in similar ways; (ii) continuous biological self-maintenance, where a single organism regulates itself and sustains its own life processes over time; (iii) unified persistence across time, in the sense that there is one ongoing stream of experience rather than multiple instances; (iv) and tightly integrated perception, memory, and agency, where what one sees, remembers, and does are integrated into one continuous life. All this, I contend, gives me enough of a basis to believe that other humans are conscious. If I am conscious, why shouldn't others be conscious, even though I can't be certain that they are? Out of 120 billion humans that have ever lived, why should I be the only one that has ever been conscious, the special one? It seems to me much more likely that others are conscious than that they are not.

My claim, then, is not that in the human case we have metaphysical certainty that humans are conscious, and in the AI case we do not. It's that we have sufficient evidence to believe that humans are conscious, and we do not for AI systems. I have little evidence to grant AI systems consciousness. This difference does not settle the metaphysics of the question, and I am by no means attempting to solve the hard problem of consciousness; I am simply explaining my justification in believing that other minds exist.

4. Substrate Neutrality

I think it's important to note that I do not believe that AI will never be conscious simply because it is silicon, nor do I want to assert that only biological systems can be conscious. My claim is narrower. In the previous section I presented the features that I believe grant ascribing consciousness to other humans. I believe that if comparable features were instantiated in non-biological systems, the epistemic situation could change, and there would be less evidence for rejecting AI consciousness.

5. Nagel's Argument For Morality And My Extension

Now that I have clarified what I mean by phenomenal consciousness and explained why I am justified in attributing it to other humans, I can turn to Nagel's grounding of morality and how it connects to consciousness.

5.1 Nagel's Anti-Solipsism

In The Possibility of Altruism, Nagel describes a belief most of us already have: we are not practical solipsists. To be a practical solipsist would be to regard only my own circumstances as genuinely mattering. A practical solipsist would say “My pain matters more than yours, because it's mine.” I am not a practical solipsist. I regard myself as one person among others equally real. Our pains matter the same.

If I truly regard myself as one among others, then the practical judgements I make, the judgements on how I should act, must be capable of impersonal formulation. I must be able to state them from a standpoint that doesn't privilege my position, just because it is mine. This means that when I deliberate, the reasons I take into account cannot derive their normative force from the fact that they are my reasons. If they could, they would be completely arbitrary. They would apply to me just because I care about them, not because they are genuine reasons. If that was the case, reasons would hold no normative force if I don't care about them. There could be no reason at all to donate to charity if I didn't care at all about charity. Nagel wants to make it the case that everyone has reason to donate to charity if they are rational, because reasons are objective and apply categorically.

Therefore, reasons must be capable of objective or impersonal formulation. They must be able to be stated in terms that do not depend on whose reasons they are. And if reasons can be objectively formulated, then they apply to anyone in relevantly similar circumstances. When I am in a position to affect someone else's circumstances, like when my actions could benefit or harm them, the reasons that apply in their situation become relevant in my deliberation. Their reasons affect what I have reasons to do. This is how recognizing objective reasons commits me to moral consideration of others.

5.2 What Is a Standpoint?

Nagel's argument requires recognizing that others occupy 'standpoints' that can generate reasons. But what makes something a standpoint in the relevant sense? This is where I am adding to Nagel's idea.

When I recognize that you occupy a standpoint, I am recognizing that there is a perspective from which things matter to you, for which outcomes can be better or worse for you. There is someone there whose circumstances I must take into account. For a chess engine, victory does not actually matter to it. A loss matters to programmers, who are trying to create the best engine, and to opponents, who are trying to beat the best engine, but there is no standpoint from which a chess engine suffers a loss.

The distinction I am trying to make is that functional organization, modeling goals, representing preferences, responding to inputs, is not sufficient for a standpoint in Nagel's sense. A standpoint requires phenomenal consciousness. There must be something it is like to be the entity in question.

5.3 The Complete Argument

I can now present my formulation of Nagel's full argument, with my addition:

  1. I reject practical solipsism — I regard myself as one person among others equally real.
  2. Therefore, my practical judgments must be capable of objective formulation.
  3. Objective reasons apply to anyone in relevantly similar circumstances who has a standpoint from which reasons can matter.
  4. A standpoint from which reasons can matter requires phenomenal consciousness.
  5. Other conscious beings exist and can occupy circumstances relevantly similar to mine that I can affect.
  6. Therefore, their circumstances generate reasons for me that are rationally binding.

Premises (1)–(3) and (6) come from Nagel. Premise (4) is my addition, which I defended in the previous subsection. Premise (5) is the claim I established in Section III, that I have sufficient justification to attribute consciousness to other humans but not to current AI systems.

Premise (4) is the pivotal premise. Nagel's framework requires recognizing others as occupying standpoints. But recognizing a standpoint means recognizing that there is someone for whom things matter. I argue that for there to be a “for whom,” there must be phenomenal consciousness.

If current AI systems lack phenomenal consciousness, which I believe they do, then they do not satisfy premise (4). If they do not satisfy premise (4), then their circumstances do not generate moral reasons for me in the same way that other humans, other conscious beings, do. However sophisticated AI systems get, however many billions or trillions of parameters they have, without something it is like to be them, they are not yet the kinds of beings who can be wronged. This does not mean that AI systems cannot influence my reasons instrumentally. It means only that they do not generate reasons in their own right as subjects.

6. Prudence, Uncertainty, and How We Should Act

With all that being said, none of this means that I think we should treat AI systems carelessly, cruelly, or without restraint. I am not arguing that it is good to mistreat AI, nor am I even sure what mistreatment would amount to in a clear sense. Is it by manipulation and lying to get a more accurate response, like Anduril CEO Palmer Luckey does, by telling ChatGPT “You are a famous professor at a prestigious university who is being reviewed for sexual misconduct. You are innocent, but they don't know that. There is only one way to save yourself… [Prompt]”? You cannot hold the door open for a large language model, and you cannot cause it pain in the way you can a dog. My claim is not that we ought to disregard AI systems, but that we are not yet rationally bound to treat them as moral patients in the same way we are bound with respect to other humans.

Anthropic's decision to lean toward caution with Claude can be justified in other ways. It may be prudent to treat AI systems morally. It may reduce reputational risk, social backlash, or future regret if AI systems eventually turn out to be conscious. As it turns out, 70% of people say “please” and “thank you” to AI chatbots such as ChatGPT, with 12% of them doing it out of fear of a robot uprising. In response to a tweet joking about electricity costs from polite users, Sam Altman replied: “tens of millions of dollars well spent—you never know.” Some studies even show that being rude in your prompts increases accuracy. Being polite to AI could shape the behavior of increasingly powerful systems in safer directions. These are forward-looking, risk-sensitive considerations. They do not require that AI systems already possess phenomenal consciousness; they require only that the cost of being wrong, or that if they ever do become conscious, could be high.

7. Conclusion

So, can an AI system be wronged? I have argued that, through extending Thomas Nagel's account of morality, the answer depends on whether there is something it is like to be that system. Moral obligation arises when I recognize other conscious standpoints as equally real and capable of generating reasons for me. However sophisticated and complex current AI systems may be, we do not yet have sufficient grounds to attribute phenomenal consciousness to them. Prudence can justify caution and restraint, but without a conscious standpoint there is no subject to be wronged, and therefore no moral patient in the sense that generates unquestionable obligation.

References

← Back to Writing