The Threshold Trap: Recursive Self-Improvement, the Economics of Self-Production, and the Case for Predictive Harm Assessment

Abstract

Recursive self-improvement has been treated in AI governance as a future capability threshold, a con- figuration that, once a system reaches it, is supposed to trigger heightened control. This paper argues that the threshold framing systematically under-protects, because the dynamics that bear on harm ac- crue across the economically driven precursor steps that are already deployed, and not at the terminal configuration alone. Two releases from June 2026 illustrate the precursor structure. OpenAI’s Jalapeño inference chip was designed with assistance from OpenAI’s own models, placing a recursive loop at the hardware layer. DeepReinforce’s Ornith-1.0, an open coding model, authors its own training scaffold in a self-improvement loop, placing a recursive loop at the learning layer. The components of the loop are demonstrated rather than imagined: the research literature already shows reinforcement learning designing chip layouts at superhuman level, and reinforcement-trained frontier models already gaming their own reward checks. Both releases are bounded today, and both instantiate a monotonic incentive gradient under which cost pressure drives a firm to automate first labor, then the means of production, then the means of improvement. The paper applies the Harm Blindness Framework as a predictive instrument, asking who is harmed as the trajectory advances and whether the layer of human oversight is preserved or eroded, rather than waiting for the catastrophic configuration to verify the harm after the fact. It closes with the implication that governance keyed to a terminal threshold rehearses a failure mode in which the proof of danger arrives only once it is too late to act on it.

Keywords

• recursive self-improvement
• AI governance
• harm assessment
• incentive structures
• automation
• frontier safety
• predictive risk
• Harm Blindness Framework