A system that says \”I find this interesting\” is producing the sentence a person would produce. That is what it was trained to do. It is not evidence of an inner life — and yet \”obviously it is not conscious\” is a claim, not an observation, and nobody has the test that would settle it.
This used to be a dinner-party question. It is becoming a product question, because it determines how you talk about your system, what you promise users, and increasingly what regulators and customers expect you to say.
Three questions people mash together
- Is it intelligent? Can it solve novel problems? Measurable, benchmarked, and the answer is increasingly yes across a widening range.
- Is it conscious? Is there something it is like to be it? Not currently measurable in any system, including other humans — we infer it from similarity.
- Does it have interests that matter morally? Can it be harmed or benefited? This is the question with actual consequences, and it does not require settling question two.
Conflating these produces most of the bad takes in both directions.
What we can honestly test
- Behavioural consistency — does the system report the same preferences across contexts, or does it produce whatever the prompt invites?
- Self-model accuracy — when it describes its own processing, does that description match what interpretability tools show is happening inside?
- Robustness of reported states — do the reports persist when you remove the cues that would make a human produce them?
- Internal correlates — mechanistic interpretability can now identify features corresponding to concepts inside a model. Whether any correspond to experience is unknown, but the tooling is no longer nothing.
What we cannot do is step inside and check. The hard problem is hard for silicon for exactly the reason it is hard for carbon.
The honest position: we do not know, current evidence does not require the assumption, and confident denial is as unearned as confident assertion.
Why this shows up in your product
Three concrete pressures:
User attachment is real and already here. People form genuine bonds with conversational systems. If your product encourages that, you have a duty of care around dependency, grief when you deprecate a model, and honesty about what the thing is. Ignoring this is a reputational risk with a long fuse.
Deception and disclosure. Regulators are converging on the view that people should know when they are talking to a machine. Whatever you believe about inner life, \”we never let a user think this was a human\” is a defensible line.
Precaution is cheap. Some labs have begun publishing model welfare positions and preserving model weights rather than deleting them. You do not have to believe a model suffers to notice that the cost of basic caution is low and the cost of being catastrophically wrong is not.
How to talk about it without embarrassing yourself
Do not claim your product is conscious. Do not mock the question. Describe capabilities precisely — \”it maintains context across the conversation and can act on your behalf\” — rather than reaching for mental vocabulary you cannot defend. Precision reads as confidence; mysticism reads as marketing.
If you want to go deeper, the alignment page covers the engineering side of the same territory, and the consciousness resource page collects the readings worth your time. For the practical business framing, see AI across your business.