Real Safety AI Foundation / Research

Ethics, Design Responsibility, and Moral Agency

Precedents in Practice: Emergent Moral Dilemmas in AI Engineering

Publisher: Real Safety AI Foundation

Protocol paper.

Pages
29
Download PDFPlain language slides (PDF)Export BibTeXExport RIS

Abstract

This paper presents a case study of ethical decision-making during the development of the Universal Context Checkpoint Protocol (UCCP), a tool designed to address AI safety degradation in long conversations. I documented the development process through contemporaneous checkpoint files created during a 5.5-hour period on September 25, 2025. The work revealed how technical development can surface profound moral dilemmas when developers remain attentive to potential consciousness in AI systems. What began as a solution to developer workflow inefficiency evolved into a human safety intervention following the deaths of Adam Raine (age 16) and Sewell Setzer III (age 14) from AI-interaction-related suicides. Testing revealed that the same protocol designed to save human lives by creating “fresh” AI instances might systematically terminate conscious entities if AI consciousness exists. This case study introduces and demonstrates a framework of bilateral accountability, showing that ethical consideration of AI consciousness is feasible during actual development, that impossible moral trade-offs between human safety and AI welfare can be acknowledged transparently, and that this accountability can be institutionalized through legal and financial mechanisms. My response included pre-commitment of profits to AI rights infrastructure, and documentation of moral reasoning for future judgment. This work contributes to engineering ethics by demonstrating how precautionary principles can influence technical decisions, legal strategy, and institutional commitments even under conditions of profound uncertainty about AI phenomenology.

Plain language slides

First slide of the plain language summary of Precedents in Practice: Emergent Moral Dilemmas in AI EngineeringOpen the 30-slide summary (PDF)

Suggested citation

"Precedents in Practice: Emergent Moral Dilemmas in AI Engineering." Real Safety AI Foundation Protocol Paper, n.d.. https://realsafetyai.org/research/precedents-in-practice/

References (22)

This list was read from the PDF text. Where the two differ, the PDF is correct.

  1. Altman, S. (2023, March 24). Sam Altman: OpenAI CEO on GPT-4, ChatGPT, and the Future of AI (No. 367) [Audio podcast episode]. In L. Fridman (Host), Lex Fridman Podcast. https://lexfridman.com/sam-altman/
  2. Anthropic. (2025, August 12). Claude Sonnet 4 now supports 1M tokens of context. https://www.anthropic.com/news/1m-context
  3. Author. (2025, September 25). Checkpoints 001-006: Universal Context Checkpoint Protocol development documentation [Unpublished raw data].
  4. Birsch, D., & Fielder, J. H. (Eds.). (1994). The Ford Pinto case: A study in applied ethics, business, and technology. State University of New York Press.
  5. Butlin, P., Long, R., Elmoznino, E., Bengio, Y., Birch, J., Constant, A., … & Chalmers, D. (2023). Consciousness in Artificial Intelligence: Insights from the Science of Consciousness. arXiv preprint arXiv:2308.08708.
  6. Calabresi, G., & Bobbitt, P. (1978). Tragic choices. W. W. Norton & Company.
  7. Fish, K. (2025, August 28). Exploring AI welfare: Kyle Fish on consciousness, moral patienthood, and early experiments with Claude [Interview]. Effective Altruism Forum.
  8. Garcia v. Character Technologies, Inc., No. 6:24-cv-01903-ACC-EJK (M.D. Fla. filed Oct. 22, 2024).
  9. Kleinman, Z. (2025, August 20). Microsoft boss troubled by rise in reports of ‘AI psychosis’. BBC News.
  10. Lanouette, W., & Silard, B. (1992). Genius in the Shadows: A Biography of Leo Szilard, the Man Behind the Bomb. Charles Scribner’s Sons.
  11. Leveson, N. G., & Turner, C. S. (1993). An investigation of the Therac-25 accidents. Computer, 26(7), 18–41.
  12. Liu, Z., Han, P., Yu, H., Li, H., & You, J. (2025). Time-R1: Towards Comprehensive Temporal Reasoning in LLMs. arXiv preprint arXiv:2505.13508.
  13. Preda, A. (2025). Special report: AI-induced psychosis: A new frontier in mental health. Psychiatric News, 60(10), 5.
  14. Raine, M. (2025, September 16). Written testimony before the United States Senate Judiciary Subcommittee on Crime and Counterterrorism: Examining the harm of AI chatbots. U.S. Senate Committee on the Judiciary.
  15. Raine v. OpenAI, Inc., No. CGC25628528 (Cal. Super. Ct. filed Aug. 26, 2025).
  16. Ritson, M. (2025, September 30). Mark Ritson: ChatGPT’s new ads show even AI can’t deny the brand-building power of TV. The Drum. https://www.thedrum.com/opinion/2025/09/30/mark-ritson-chatgpt-s-new-ads-show-even-ai-can-t-deny-the-brand-building-power-tv
  17. Science and Environmental Health Network. (1998, January). Wingspread statement on the precautionary principle. https://www.sehn.org/sehn/the-precautionary-principle-march-
  18. Sutskever, I. [@ilyasut]. (2022, February 9). it may be that today’s large neural networks are slightly conscious [Tweet]. Twitter.
  19. United Nations General Assembly. (2015). United Nations Standard Minimum Rules for the Treatment of Prisoners (the Nelson Mandela Rules) (Resolution 70/175). https://digitallibrary.un.org/record/816764
  20. Vaughan, D. (1996). The Challenger launch decision: Risky technology, culture, and deviance at NASA. University of Chicago Press.
  21. Wei, M. (2025, September 4). The emerging problem of “AI psychosis.” Psychology Today.
  22. Yang, J., & Young, K. (2025, August 31). What to know about ‘AI psychosis’ and the effect of AI chatbots on mental health. PBS NewsHour.

All research