Children, Schools, and Youth Policy
Children as Canaries: The Pediatric Sentinel Effect in Algorithmic Harm
- Travis Gilly, Real Safety AI Foundation
Publisher: Real Safety AI Foundation
Working paper. Not peer reviewed.
- Written
- December 2025
- Pages
- 5
Abstract
Children function as sentinel populations for algorithmic harm in the same way they serve as early indicators of environmental toxicity in public health research. This paper introduces the Pediatric Sentinel Effect in Algorithmic Harm, a framework proposing that harms experienced by children from AI and algorithmic systems reliably precede recognition of those same harms in adult populations. Drawing on analysis of 40 source documents (including UNICEF regional assessments, the UN Convention on the Rights of the Child, and the AI Incidents Database) comprising over 7,300 extracted harm terms, we identify three domains where the sentinel effect is empirically observable: biometric surveillance (school facial recognition deployment preceded adult wrongful arrest cases by eight months), predictive risk systems (child welfare algorithms established precedents for administrative data misuse now spreading to adult contexts), and recommender systems (documented mental health impacts on adolescents preceded clinical recognition of algorithmic addiction in adults). We propose three mechanisms driving this effect: mandatory institutional exposure reducing children's capacity to opt out, cognitive developmental factors increasing susceptibility to algorithmic manipulation, and paradoxical data vulnerability wherein the most legally protected population generates the most behavioral data. The framework has limitations in economic and labor domains where adults experience harms first. We argue that child-focused AI safety testing should be treated as a leading indicator methodology for population-level risk assessment, and that existing pediatric sentinel frameworks from toxicology and environmental health offer immediate methodological templates for AI governance.
Plain language slides
Open the 19-slide summary (PDF)Suggested citation
Gilly, Travis. "Children as Canaries: The Pediatric Sentinel Effect in Algorithmic Harm." Real Safety AI Foundation Working Paper, December 2025. https://realsafetyai.org/research/children-as-canaries-working-paper/
Other versions
This paper is also posted on SSRN.
References (10)
This list was read from the PDF text. Where the two differ, the PDF is correct.
- Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016, May 23). Machine Bias. ProPublica. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
- Equal Employment Opportunity Commission. (2023, August 9). iTutorGroup to Pay $365,000 to Settle EEOC Discriminatory Hiring Suit [Press release]. https://www.eeoc.gov/newsroom/itutorgroup-pay-365000-settle-eeoc-discriminatory-hiring-suit
- Eubanks, V. (2018). Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. St. Martin's Press.
- 5Rights Foundation. (2021). But How Do They Know It Is a Child? Age Assurance in the Digital World. https://5rightsfoundation.com/
- Hill, K. (2020, June 24). Wrongfully Accused by an Algorithm. New York Times. https://www.nytimes.com/2020/06/24/technology/facial-recognition-arrest.html
- Mobley v. Workday, Inc., No. 23-cv-00770-RFL (N.D. Cal. July 12, 2024) (order denying motion to dismiss).
- New York State Senate. (2020). Senate Bill S5140B: An act to amend the education law, in relation to the use of biometric identifying technology. Signed December 22, 2020, Chapter 349. https://www.nysenate.gov/legislation/bills/2019/s5140
- UNICEF Office of Research - Innocenti. (2020). Policy Guidance on AI for Children. https://www.unicef.org/globalinsight/reports/policy-guidance-ai-children
- Weiss, E. (2022, July 6). NYCLU Finds New York Schools Not Sticking to Facial Recognition Ban. ID Tech Wire. https://idtechwire.com/nyclu-finds-new-york-schools-not-sticking-facial-recognition-ban-070602/
- Wells, G., Horwitz, J., & Seetharaman, D. (2021, September 14). Facebook Knows Instagram Is Toxic for Teen Girls, Company Documents Show. Wall Street Journal. https://www.wsj.com/articles/facebook-knows-instagram-is-toxic-for-teen-girls-company-documents-show-11631620739