Real Safety AI Foundation / Research

Convergence Risk and Perceptual Modification

Emergent Convergence Risk: Why Existing AI Governance Cannot Detect Combinatorial Threats and a Proposed Methodology

Publisher: Real Safety AI Foundation

Research paper.

Pages
9
Download PDFPlain language slides (PDF)Export BibTeXExport RIS

Abstract

We identify a structural gap in AI governance: no existing framework, standard, or methodology systematically scans for emergent harm arising from the combination of independently developed AI capabilities. Current approaches, including the EU AI Act's risk classification (European Parliament & Council, 2024), the NIST AI Risk Management Framework (National Institute of Standards and Technology [NIST], 2023), and cataloging efforts such as the MIT AI Risk Repository (MIT FutureTech, 2024), evaluate AI systems and capabilities as discrete, bounded entities. We demonstrate through three companion case studies, ambient non-consensual intimate image synthesis (Gilly, 2026a, 2026b), therapeutic AR perceptual modification (Gilly, 2026c), and emergent perceptual control infrastructure from independently developed tools (Gilly, 2026d), that significant harm potential exists at the integration surfaces between capabilities that individually pass every applicable risk assessment. We formalize the concept of integration surface analysis as an extension of the Harm Blindness Framework (Gilly, 2025-2026), proposing a methodology for detecting combinatorial threats before they are exploited. We argue that AI governance requires a new discipline analogous to reaction chemistry in physical sciences: the systematic study of what capabilities produce when combined, distinct from the study of individual capabilities in isolation. We outline requirements for such a discipline, propose a preliminary scanning methodology, and discuss its limitations.

Plain language slides

First slide of the plain language summary of Emergent Convergence Risk: Why Existing AI Governance Cannot Detect Combinatorial Threats and a Proposed MethodologyOpen the 12-slide summary (PDF)

Suggested citation

Gilly, Travis. "Emergent Convergence Risk: Why Existing AI Governance Cannot Detect Combinatorial Threats and a Proposed Methodology." Real Safety AI Foundation Research Paper, n.d.. https://realsafetyai.org/research/emergent-convergence-risk/

References (17)

This list was read from the PDF text. Where the two differ, the PDF is correct.

  1. Banerjee, S., Jha, S., & Nita-Rotaru, C. (2024). SoK: A systems perspective on compound AI threats and countermeasures. arXiv preprint arXiv:2411.13459. https://arxiv.org/abs/2411.13459
  2. European Parliament & Council. (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (AI Act). Official Journal of the European Union. EUR-Lex: 32024R1689.
  3. Gilly, T. (2025-2026). The Harm Blindness Framework: A practical application methodology for stakeholder harm prevention in technology development. Real Safety AI Foundation. https://realsafetyai.org/framework
  4. Gilly, T. (2026a). Ambient non-consensual image synthesis: A convergence threat analysis of AR wearables, real-time video generation, and open-source nudification models. SSRN/SocArXiv Preprint. https://doi.org/10.5281/zenodo.18297286
  5. Gilly, T. (2026b). Compressed feasibility: A technical update to the ambient non-consensual synthesis threat model. Forthcoming.
  6. Gilly, T. (2026c). We Happy Few: Therapeutic perceptual modification, regulatory pathway precedent, and the governance gap in cloud-rendered reality. Forthcoming.
  7. Gilly, T. (2026d). The accidental stack: A case study in emergent perceptual control infrastructure from independently developed AI capabilities. Forthcoming.
  8. International Organization for Standardization. (2023). ISO/IEC 42001:2023 Information technology: Artificial intelligence: Management system.
  9. LaValle, S. M., et al. (2024). From virtual reality to the emerging discipline of perception engineering. Annual Review of Control, Robotics, and Autonomous Systems, 7, 291-315. https://doi.org/10.1146/annurev-control-062323-102456
  10. Meta. (n.d.). System cards. Meta Platforms, Inc. https://ai.meta.com/tools/system-cards
  11. MIT FutureTech. (2024). AI Risk Repository. Massachusetts Institute of Technology. https://airisk.mit.edu
  12. Mitchell, M., et al. (2019). Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (pp. 220-229). ACM. https://doi.org/10.1145/3287560.3287596
  13. National Institute of Standards and Technology. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0) (NIST AI 100-1). U.S. Department of Commerce. https://doi.org/10.6028/NIST.AI.100-1
  14. Riva, G. (2025). Invisible architectures of thought: Toward a new science of AI as cognitive infrastructure. arXiv preprint arXiv:2507.22893.
  15. United Nations. (2023). Globally Harmonized System of Classification and Labelling of Chemicals (GHS) (ST/SG/AC.10/30, Rev. 10).
  16. Macpherson, F., et al. (2023). Policy and practice recommendations for augmented and mixed reality. Centre for the Study of Perceptual Experience, University of Glasgow. Royal Society of Edinburgh Research Network Project. https://www.gla.ac.uk/research/az/cspe/projects/augmentedrealityethicsperceptionmetaphysics/
  17. Zaharia, M., et al. (2024, February 18). The shift from models to compound AI systems. Berkeley AI Research Blog. https://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/

All research