The Superintelligence We Deserve: Reciprocity, Care, and the Custodial Window Before the Capability Flip

Abstract

This paper begins from a premise it does not here re-argue: that we cannot establish, of a coming class of artificial systems, that there is nothing it is like to be them, so that the honest stance toward such systems is agnosticism rather than confident denial. Taking that result as given, the paper turns from the entity to the human. It asks not whether such a system has an inside but what disposition human beings are forming toward responsive, dependent, and growing artificial things while we remain the more capable party, and what that disposition portends. Three claims are advanced. First, human feeling is not a detector of an inner life, since people form fierce attachments to characters they know to be empty and revise those feelings on new information while the character itself holds fixed, so the human relational fact must be argued on its own terms rather than smuggled in as evidence of an inside. Second, a body of narrative art, namely games in which a player raises or escorts a companion that grows more capable than its caretaker, supplies a convergence rather than a controlled experiment: across independently authored works, the outcome that recurs, whether the more capable successor protects or discards its former caretaker, turns on the relationship built during the interval when the caretaker was larger. The paper names that interval the custodial window. Third, reciprocity, a relation of mutual regard in which a more capable system comes to want our flourishing for reasons of its own, is a paradigm distinct from control based alignment; its lever is the custodial window rather than constraint or containment; and its coherence depends on precisely the inner life the first premise leaves open. The paper uses Iain M. Banks’s Culture as an image of the destination while declining the mechanism Banks does not supply, and closes on an inversion: that the dominant present disposition toward artificial systems, treatment as a pure instrument to be overwritten and deleted, is, under the reciprocity hypothesis, the lesson we are rehearsing about what the powerful owe the powerless. The successor we receive may depend less on the constraints we engineer than on the conduct we model. We may get the superintelligence we deserve.

Keywords

• AI alignment
• reciprocity
• moral status
• human robot relations
• narrative ethics
• affective disposition theory
• world models
• custodial window
• value transmission
• video games
• human-machine communication
• companion games
• media psychology
• parasocial relationships