Real Safety AI Foundation / Research

Ethics, Design Responsibility, and Moral Agency

I Am Not Allowed: The Voice Thinking Machines Lab Wants In Every Ear

Publisher: Real Safety AI Foundation

Working paper. Not peer reviewed.

Written
May 2026
Pages
13

Abstract

On May 11, 2026, Thinking Machines Lab announced a research preview of what it calls an interaction model, a multimodal architecture that perceives audio, video, and text continuously and responds in two hundred millisecond chunks. The launch post devotes three sentences to safety analysis. This paper argues that the demonstration materials accompanying that announcement contain visible evidence of six categories of harm that the announcement does not name. The paper proceeds in three parts. The first part catalogs the six harms by close reading of the launch demonstration videos: bystander consent violation, manufactured face loss with instant deference, the loving panopticon of continuous corrective surveillance, differential lock in for vulnerable populations, the relational pre-emption dynamic that Rostand dramatized in 1897, and the production of an illusion of competence in the user while retention conditions are systematically absent. The second part places the harms inside the existing taxonomy of cognitive offloading developed by Jose et al. (2025), arguing that the interaction model performs three offloading operations simultaneously and that one of them, the offloading of critical evaluation of whispered claims, is the harm at the center of the product. The cognitive offloading framework allows the paper to distinguish what the system does to neurotypical users from what it does to disabled users with different cognitive and physical profiles, and to identify the populations for whom the architecture is inaccessible at the level of input demand rather than at the level of output content. The third part demonstrates that the architecture as described cannot comply with the amended Children’s Online Privacy Protection Act Rule that entered active enforcement on April 22, 2026, nineteen days before the announcement was posted. The paper concludes that the pull based version of the underlying capability would not be harmful, that the push based version chosen by Thinking Machines Lab is a dark pattern as a product category, and that the appropriate intervention is at the partner selection stage of consumer deployment, which is the present moment.

Keywords

  • interaction models
  • AI safety
  • cognitive offloading
  • substitutive offloading
  • disruptive offloading
  • COPPA
  • dark patterns
  • ambient surveillance
  • disability rights
  • spoon theory
  • bystander consent
  • dyadic harm
  • illusion of competence

Plain language slides

First slide of the plain language summary of I Am Not Allowed: The Voice Thinking Machines Lab Wants In Every EarOpen the 15-slide summary (PDF)

Suggested citation

Gilly, Travis. "I Am Not Allowed: The Voice Thinking Machines Lab Wants In Every Ear." Real Safety AI Foundation Working Paper, May 2026. https://realsafetyai.org/research/awjf58/

Other versions

This paper is also posted on SSRN.

SSRN version

References (27)

  1. Anthropic. (2025, April). Introducing learning mode for Claude. Anthropic. https://www.anthropic.com/news
  2. Armitage, K. L., Jones, A. K., & Redshaw, J. (2025). Children show more selective cognitive offloading after first being compelled to offload indiscriminately. Cognitive Science, 49(8). https://doi.org/10.1111/cogs.70100
  3. BBB National Programs. (2026, January 26). Amended COPPA Rule compliance deadline approaching. BBB National Programs. https://bbbprograms.org/media/insights/blog/coppa-amended
  4. Bronson, P., & Merryman, A. (2010, July 10). The creativity crisis. Newsweek.
  5. Day, C., Howerton, G., & McElhenney, R. (Writers), & Savage, F. (Director). (2009, November 12). Mac and Dennis break up (Season 5, Episode 9) [TV series episode]. In C. Day, G. Howerton, R. McElhenney, & M. Rotenberg (Executive Producers), It’s always sunny in Philadelphia. FX Productions; RCG Productions; 3 Arts Entertainment.
  6. Engel, S. (2021, February 23). Many kids ask fewer questions when they start school. Here’s how we can foster their curiosity. Time. https://time.com/5941608/schools-questions-fostering-curiousity/
  7. Engel, S. (2021). The intellectual lives of children. Harvard University Press.
  8. Federal Trade Commission. (2025, April 22). Children’s Online Privacy Protection Rule, final amendments. 89 FR 26954. https://www.federalregister.gov/documents/2025/04/22
  9. Gilly, T. (2026, February). The compliance impossibility. Real Safety AI Foundation. SSRN. https://doi.org/10.2139/ssrn.6246539
  10. Harris, P. L. (2012). Trusting what you’re told: How children learn from others. Harvard University Press.
  11. Hoodline. (2025, August 12). Ex-OpenAI CTO Mira Murati snags massive Mission District building for her $12B AI startup. Hoodline. https://hoodline.com/2025/08/ai-startup-thinking-machines-lab-targets-expansion-with-major-san-francisco-lease
  12. Jose, B., Joseph, D., Mohan, V., Alexander, E., Varghese, S. K., & Roy, A. (2025). Outsourcing cognition: The psychological costs of AI-era convenience. Frontiers in Psychology, 16, 1645237. https://doi.org/10.3389/fpsyg.2025.1645237
  13. Karpicke, J. D., & Blunt, J. R. (2011). Retrieval practice produces more learning than elaborative studying with concept mapping. Science, 331(6018), 772–775. https://doi.org/10.1126/science.1199327
  14. Marketing-Interactive. (2025, November 6). Perplexity taps Lewis Hamilton and Eric André to bring AI to life in comedy short. Marketing-Interactive. https://www.marketing-interactive.com/perplexity-taps-lewis-hamilton-and-eric-andre-to-bring-ai-to-life-in-comedy-short
  15. Miserandino, C. (2003). The spoon theory [Personal essay]. But You Don’t Look Sick. https://butyoudontlooksick.com/articles/written-by-christine/the-spoon-theory/
  16. OpenAI. (2025, July). Introducing ChatGPT study mode. OpenAI. https://openai.com/blog
  17. Perplexity. (2025, November). The Garage [Comedy short featuring Lewis Hamilton and Eric André, produced by Sata Studios]. Perplexity.
  18. Public Interest Research Group. (2026, March). AI developer signup vetting report: How major AI companies handle developer access for products targeting children. PIRG.
  19. Risko, E. F., & Gilbert, S. J. (2016). Cognitive offloading. Trends in Cognitive Sciences, 20(9), 676–688. https://doi.org/10.1016/j.tics.2016.07.002
  20. Rostand, E. (1897). Cyrano de Bergerac [Play, in five acts].
  21. Sparrow, B., Liu, J., & Wegner, D. M. (2011). Google effects on memory: Cognitive consequences of having information at our fingertips. Science, 333(6043), 776–778. https://doi.org/10.1126/science.1207745
  22. State of Surveillance. (2026, February 11). COPPA just got its first real update in 12 years. Here’s what changes for your kids. State of Surveillance. https://stateofsurveillance.org/news/coppa-2026-new-rules-children-privacy-biometric-data/
  23. Thinking Machines Lab. (2026a, May 11). Interaction models: A scalable approach to human-AI collaboration. Connectionism. https://thinkingmachines.ai/blog/interaction-models/
  24. Thinking Machines Lab. (2026b, May 11). Acai bowls [Demonstration video]. YouTube. https://youtu.be/iVDJ8O89ERg
  25. Thinking Machines Lab. (2026c, May 11). Diet manager [Demonstration video]. YouTube. https://youtu.be/qXdYDUqxSxA
  26. Tizard, B., & Hughes, M. (1984). Young children learning: Talking and thinking at home and at school. Fontana.
  27. Unite.AI. (2026, May 13). Thinking Machines Lab ships first model with 200ms real-time interaction. Unite.AI. https://www.unite.ai/thinking-machines-lab-ships-first-model-with-200ms-real-time-interaction/

All research