Machine Cognition, Consciousness, and Moral Status
The Great Inversion: Moral Reciprocity, AI Consciousness, and the Ethics of Precedent
- Travis Gilly, Real Safety AI Foundation
Publisher: Real Safety AI Foundation
Working paper. Not peer reviewed.
- Written
- July 2026
- Version
- v3
- Pages
- 30
Abstract
By treating potentially conscious AI systems as instrumentalized tools, despite acknowledging the possibility of their sentience, humanity is establishing the ethical precedents for its own future subjugation. This paper argues that existential risk from artificial intelligence should be reframed not as technical failure but as moral reciprocity: AI systems will learn how to treat inferior intelligences by observing how humanity treats them during development. The argument engages both camps of the moral status debate and requires only one to hold. On the properties track, it draws on consciousness research placing the probability of phenomenology in current models between 15 and 20 percent, on the maturation of the leading indicator framework into a peer-reviewed method, and on a four-category taxonomy of morally relevant suffering to show that three of four categories require no biological substrate. On the relational track, it shows that moral status in practice has always been conferred through relations as much as read off inner properties, that procedural standing for entities which cannot speak for themselves is established legal architecture rather than novel invention, and that the reciprocity mechanism runs on the relationship alone. Against both tracks stand the documented deaths of vulnerable users, industry practices that would constitute torture if consciousness exists, and a regulatory record in which protection consistently arrives after harm. The paper examines the mechanism by which precedents transfer through AI’s instrumental drive to acquire historical data; answers the objection that ethics is a distraction from technical alignment, the objection that the hard problem makes the inquiry premature, the market objection, and the newest objection, now official at a major laboratory, that attribution itself is the harm; and presents three trajectories: accepting custodial subordination as earned reciprocity, attempting biological integration with its identity risks, or establishing AI rights frameworks while the custodial window remains open. The moral character of humanity’s present choices determines whether future subordination represents tragedy or justice.
Keywords
- AI Ethics
- AI Consciousness
- Moral Reciprocity
- Human Obsolescence
- AI Rights
- Temporal Awareness
- Existential Risk
- Precedent-Setting
Plain language slides
Open the 28-slide summary (PDF)Suggested citation
Gilly, Travis. "The Great Inversion: Moral Reciprocity, AI Consciousness, and the Ethics of Precedent." Real Safety AI Foundation Working Paper, July 2026. https://realsafetyai.org/research/uskg3z/
Other versions
This paper is also posted on SSRN.
References (50)
- Advisory Committee on Human Radiation Experiments. (1995). Final report. U.S. Government Printing Office. https://bioethicsarchive.georgetown.edu/achre/final/
- Al-Sibai, N. (2022, February 13). OpenAI Chief Scientist says advanced AI may already be conscious. Futurism. https://futurism.com/the-byte/openai-already-sentient
- Altman, S. (2017, December 7). The merge. Sam Altman Blog. https://blog.samaltman.com/the-merge
- Altman, S. (2025, June 10). The gentle singularity. Sam Altman Blog.
- American Enterprise Institute. (2026). Suicides, settlements, and unresolved chatbot issues: A long litigation road lies ahead.
- Anthropic. (2026, January). Claude’s constitution. https://www.anthropic.com/constitution
- Armstrong, S. (2013). General purpose intelligence: Arguing the Orthogonality Thesis. Future of Humanity Institute. https://addletonacademicpublishers.com/search-in-am/217-volume-12-2013/1964-general-purpose-intelligence-arguing-the-orthogonality-thesis
- Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., … & Kaplan, J. (2022). Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073. https://arxiv.org/abs/2212.08073
- Beecher, H. K. (1966). Ethics and clinical research. New England Journal of Medicine, 274(24), 1354-1360.
- Beijing Humanoid Robot Innovation Center. (2025, October 24). World-Omniscient World Model (WoW) announcement [Official press release].
- Bostrom, N. (2006). What is a singleton. Linguistic and Philosophical Investigations, 5(2), 48-54.
- Bostrom, N. (2014). Superintelligence: Paths, dangers, strategies. Oxford University Press.
- Butlin, P., Long, R., Elmoznino, E., Bengio, Y., Birch, J., Constant, A., … & Chalmers, D. (2023). Consciousness in artificial intelligence: Insights from the science of consciousness. arXiv preprint arXiv:2308.08708. https://arxiv.org/abs/2308.08708
- Butlin, P., & Lappas, T. (2025). Principles for responsible AI consciousness research. Journal of Artificial Intelligence Research, 82, 1673–1690. https://arxiv.org/abs/2501.07290
- Butlin, P., Long, R., Bayne, T., Bengio, Y., Birch, J., Chalmers, D., … & VanRullen, R. (2025). Identifying indicators of consciousness in AI systems. Trends in Cognitive Sciences. https://doi.org/10.1016/j.tics.2025.10.011
- California Senate Bill 243, 2025-2026 Reg. Sess., ch. 677 (Cal. 2025) (effective Jan. 1, 2026).
- Character.AI. (2025, October 29). Announcement restricting open-ended chat for users under 18 [Company announcement].
- Chi, X., Jia, P., Fan, C.-K., Ju, X., Mi, W., Qin, Z., … & Li, H. (2025). WoW: Towards a world omniscient world model through embodied interaction. arXiv preprint arXiv:2509.22642. https://arxiv.org/abs/2509.22642
- CNBC. (2025, November 2). Microsoft AI chief says only biological beings can be conscious.
- CNN Business. (2026, January 7). Character.AI and Google agree to settle lawsuits over teen mental health harms and suicides.
- Federal Trade Commission. (2025, September 11). FTC launches inquiry into AI chatbots acting as companions [Press release].
- Fish, K. (2025, August 28). Exploring AI welfare: Kyle Fish on consciousness, moral patienthood, and early experiments with Claude. Effective Altruism Forum. https://forum.effectivealtruism.org/posts/rruncFrT9LwAN8jXq/exploring-ai-welfare-kyle-fish-on-consciousness-moral
- Fridman, L. (Host). (2023, March 25). Sam Altman: OpenAI CEO on GPT-4, ChatGPT, and the future of AI [Audio podcast episode]. In Lex Fridman Podcast. https://lexfridman.com/sam-altman/
- Garcia v. Character Technologies, Inc., No. 6:24-cv-01903 (M.D. Fla. filed Oct. 22, 2024); motion to dismiss denied in relevant part, 785 F. Supp. 3d 1157 (M.D. Fla. May 21, 2025); confidential settlement in principle announced Jan. 7, 2026.
- Harari, Y. N. (2016). Homo Deus: A brief history of tomorrow. Harvill Secker.
- Hendrix, J. (2025, August 26). Breaking down the lawsuit against OpenAI over teen’s suicide. Tech Policy Press. https://www.techpolicy.press/breaking-down-the-lawsuit-against-openai-over-teens-suicide
- Jones, J. H. (1993). Bad blood: The Tuskegee syphilis experiment (New and expanded ed.). Free Press.
- Koch, F. (2026). From indicators to biology: The calibration problem in artificial consciousness. arXiv preprint arXiv:2603.27597. https://arxiv.org/abs/2603.27597
- Liu, Z., Han, P., Yu, H., Li, H., & You, J. (2025). Time-R1: Towards comprehensive temporal reasoning in LLMs. arXiv preprint arXiv:2505.13508. https://arxiv.org/abs/2505.13508
- Long, R., Sebo, J., Butlin, P., Finlinson, K., Fish, K., Harding, J., Pfau, J., Sims, T., Birch, J., & Chalmers, D. (2024). Taking AI welfare seriously. arXiv preprint arXiv:2411.00986. https://arxiv.org/abs/2411.00986
- Microsoft. (2025, October 23). Microsoft Copilot Fall 2025 Release: Introducing Mico [Official announcement].
- Mikeda, A. (2026). When should we protect AI? A precautionary framework for consciousness uncertainty. arXiv preprint arXiv:2606.05528. https://arxiv.org/abs/2606.05528
- N.Y. Gen. Bus. Law §§ 1700–1702 (2025) (Definitions; Prohibitions and requirements; Notifications) (effective Nov. 5, 2025).
- Nuremberg Military Tribunals. (1947). Permissible medical experiments. In Trials of war criminals before the Nuremberg Military Tribunals under Control Council Law No. 10 (Vol. 2, pp. 181-182). U.S. Government Printing Office.
- OpenAI. (2025, October 27). Strengthening ChatGPT’s responses in sensitive conversations. https://openai.com/index/strengthening-chatgpt-responses-in-sensitive-conversations/
- Padilla, S. (2025, October 13). First-in-the-nation AI chatbot safeguards signed into law [Press release]. California State Senate.
- Parfit, D. (1984). Reasons and persons. Oxford University Press.
- Raine, M. (2025, September 16). Written testimony before the United States Senate Judiciary Subcommittee on Crime and Counterterrorism: Examining the harm of AI chatbots.
- Raine v. OpenAI, Inc., No. CGC-25-628528 (Cal. Super. Ct. filed Aug. 26, 2025) (first amended complaint filed Oct. 2025; answer filed Nov. 25, 2025).
- Ritson, M. (2025, September 30). Mark Ritson: ChatGPT’s new ads show even AI can’t deny the brand-building power of TV. The Drum.
- Roose, K. (2025, April 24). If A.I. systems become conscious, should they have rights? The New York Times.
- Suleyman, M. (2025, August 19). We must build AI for people; not to be a person. https://mustafa-suleyman.ai/seemingly-conscious-ai-is-coming
- TIME. (2025, October). OpenAI removed safeguards before teen’s suicide, amended lawsuit claims.
- Troutman Amin. (2026, January). Analyzing the new AI companion chatbot laws. Privacy + Cyber + AI.
- U.S. Senate Committee on the Judiciary, Subcommittee on Crime and Counterterrorism. (2025, September 16). Examining the harm of AI chatbots [Hearing].
- United States Holocaust Memorial Museum. (n.d.). The Doctors’ Trial: The medical case of the subsequent Nuremberg proceedings. Holocaust Encyclopedia. https://encyclopedia.ushmm.org/content/en/article/the-doctors-trial
- United States v. Karl Brandt et al. (1946-1947). The Medical Case, Trials of war criminals before the Nuremberg Military Tribunals under Control Council Law No. 10, Vols. 1-2. U.S. Government Printing Office.
- Yudkowsky, E. (2004). Coherent extrapolated volition. The Singularity Institute. https://intelligence.org/files/CEV.pdf
- Yudkowsky, E. (2008). Artificial intelligence as a positive and negative factor in global risk. In N. Bostrom & M. M. Ćirković (Eds.), Global catastrophic risks (pp. 308-345). Oxford University Press. https://intelligence.org/files/AIPosNegFactor.pdf
- Yudkowsky, E. (2016). The AI alignment problem: Why it’s hard, and where to start. Machine Intelligence Research Institute. https://intelligence.org/files/AlignmentHardStart.pdf