Ethics, Design Responsibility, and Moral Agency
I Am Not Allowed: The Voice Thinking Machines Lab Wants In Every Ear
- Travis Gilly, Real Safety AI Foundation
Publisher: Real Safety AI Foundation
Working paper. Not peer reviewed.
- Written
- May 2026
- Pages
- 13
Abstract
On May 11, 2026, Thinking Machines Lab announced a research preview of what it calls an interaction model, a multimodal architecture that perceives audio, video, and text continuously and responds in two hundred millisecond chunks. The launch post devotes three sentences to safety analysis. This paper argues that the demonstration materials accompanying that announcement contain visible evidence of six categories of harm that the announcement does not name. The paper proceeds in three parts. The first part catalogs the six harms by close reading of the launch demonstration videos: bystander consent violation, manufactured face loss with instant deference, the loving panopticon of continuous corrective surveillance, differential lock in for vulnerable populations, the relational pre-emption dynamic that Rostand dramatized in 1897, and the production of an illusion of competence in the user while retention conditions are systematically absent. The second part places the harms inside the existing taxonomy of cognitive offloading developed by Jose et al. (2025), arguing that the interaction model performs three offloading operations simultaneously and that one of them, the offloading of critical evaluation of whispered claims, is the harm at the center of the product. The cognitive offloading framework allows the paper to distinguish what the system does to neurotypical users from what it does to disabled users with different cognitive and physical profiles, and to identify the populations for whom the architecture is inaccessible at the level of input demand rather than at the level of output content. The third part demonstrates that the architecture as described cannot comply with the amended Children’s Online Privacy Protection Act Rule that entered active enforcement on April 22, 2026, nineteen days before the announcement was posted. The paper concludes that the pull based version of the underlying capability would not be harmful, that the push based version chosen by Thinking Machines Lab is a dark pattern as a product category, and that the appropriate intervention is at the partner selection stage of consumer deployment, which is the present moment.
Keywords
- interaction models
- AI safety
- cognitive offloading
- substitutive offloading
- disruptive offloading
- COPPA
- dark patterns
- ambient surveillance
- disability rights
- spoon theory
- bystander consent
- dyadic harm
- illusion of competence
Plain language slides
Open the 15-slide summary (PDF)Suggested citation
Gilly, Travis. "I Am Not Allowed: The Voice Thinking Machines Lab Wants In Every Ear." Real Safety AI Foundation Working Paper, May 2026. https://realsafetyai.org/research/i-am-not-allowed/
Other versions
This paper is also posted on SSRN.
References (27)
- Anthropic. (2025, April). Introducing learning mode for Claude. Anthropic. https://www.anthropic.com/news
- Armitage, K. L., Jones, A. K., & Redshaw, J. (2025). Children show more selective cognitive offloading after first being compelled to offload indiscriminately. Cognitive Science, 49(8). https://doi.org/10.1111/cogs.70100
- BBB National Programs. (2026, January 26). Amended COPPA Rule compliance deadline approaching. BBB National Programs. https://bbbprograms.org/media/insights/blog/coppa-amended
- Bronson, P., & Merryman, A. (2010, July 10). The creativity crisis. Newsweek.
- Day, C., Howerton, G., & McElhenney, R. (Writers), & Savage, F. (Director). (2009, November 12). Mac and Dennis break up (Season 5, Episode 9) [TV series episode]. In C. Day, G. Howerton, R. McElhenney, & M. Rotenberg (Executive Producers), It’s always sunny in Philadelphia. FX Productions; RCG Productions; 3 Arts Entertainment.
- Engel, S. (2021, February 23). Many kids ask fewer questions when they start school. Here’s how we can foster their curiosity. Time. https://time.com/5941608/schools-questions-fostering-curiousity/
- Engel, S. (2021). The intellectual lives of children. Harvard University Press.
- Federal Trade Commission. (2025, April 22). Children’s Online Privacy Protection Rule, final amendments. 89 FR 26954. https://www.federalregister.gov/documents/2025/04/22
- Gilly, T. (2026, February). The compliance impossibility. Real Safety AI Foundation. SSRN. https://doi.org/10.2139/ssrn.6246539
- Harris, P. L. (2012). Trusting what you’re told: How children learn from others. Harvard University Press.
- Hoodline. (2025, August 12). Ex-OpenAI CTO Mira Murati snags massive Mission District building for her $12B AI startup. Hoodline. https://hoodline.com/2025/08/ai-startup-thinking-machines-lab-targets-expansion-with-major-san-francisco-lease
- Jose, B., Joseph, D., Mohan, V., Alexander, E., Varghese, S. K., & Roy, A. (2025). Outsourcing cognition: The psychological costs of AI-era convenience. Frontiers in Psychology, 16, 1645237. https://doi.org/10.3389/fpsyg.2025.1645237
- Karpicke, J. D., & Blunt, J. R. (2011). Retrieval practice produces more learning than elaborative studying with concept mapping. Science, 331(6018), 772–775. https://doi.org/10.1126/science.1199327
- Marketing-Interactive. (2025, November 6). Perplexity taps Lewis Hamilton and Eric André to bring AI to life in comedy short. Marketing-Interactive. https://www.marketing-interactive.com/perplexity-taps-lewis-hamilton-and-eric-andre-to-bring-ai-to-life-in-comedy-short
- Miserandino, C. (2003). The spoon theory [Personal essay]. But You Don’t Look Sick. https://butyoudontlooksick.com/articles/written-by-christine/the-spoon-theory/
- OpenAI. (2025, July). Introducing ChatGPT study mode. OpenAI. https://openai.com/blog
- Perplexity. (2025, November). The Garage [Comedy short featuring Lewis Hamilton and Eric André, produced by Sata Studios]. Perplexity.
- Public Interest Research Group. (2026, March). AI developer signup vetting report: How major AI companies handle developer access for products targeting children. PIRG.
- Risko, E. F., & Gilbert, S. J. (2016). Cognitive offloading. Trends in Cognitive Sciences, 20(9), 676–688. https://doi.org/10.1016/j.tics.2016.07.002
- Rostand, E. (1897). Cyrano de Bergerac [Play, in five acts].
- Sparrow, B., Liu, J., & Wegner, D. M. (2011). Google effects on memory: Cognitive consequences of having information at our fingertips. Science, 333(6043), 776–778. https://doi.org/10.1126/science.1207745
- State of Surveillance. (2026, February 11). COPPA just got its first real update in 12 years. Here’s what changes for your kids. State of Surveillance. https://stateofsurveillance.org/news/coppa-2026-new-rules-children-privacy-biometric-data/
- Thinking Machines Lab. (2026a, May 11). Interaction models: A scalable approach to human-AI collaboration. Connectionism. https://thinkingmachines.ai/blog/interaction-models/
- Thinking Machines Lab. (2026b, May 11). Acai bowls [Demonstration video]. YouTube. https://youtu.be/iVDJ8O89ERg
- Thinking Machines Lab. (2026c, May 11). Diet manager [Demonstration video]. YouTube. https://youtu.be/qXdYDUqxSxA
- Tizard, B., & Hughes, M. (1984). Young children learning: Talking and thinking at home and at school. Fontana.
- Unite.AI. (2026, May 13). Thinking Machines Lab ships first model with 200ms real-time interaction. Unite.AI. https://www.unite.ai/thinking-machines-lab-ships-first-model-with-200ms-real-time-interaction/