"The robot should pause and think when conditions degrade" is a requirement that survives design review because it sounds like one specification. It is three, they operate at different layers, and they fail independently. Conflating them produces safety architectures that look complete while leaving the hardest component unimplemented.
Separated out: maintaining the ability to stop, allocating more inference compute under uncertainty, and transferring control to a human. Only the first is an open problem, and it receives the least attention.
1. Abort reachability
Pausing presupposes that a safe stop is available from the current state. For a system mid-motion this is a substantive assumption and frequently false. A gripper partway through extracting a friction-fit block has already altered the structure's load path. Holding position is a control input with consequences, not a null action.
The property worth specifying is continuous rather than instantaneous: at every timestep, there exists a control sequence from the current state that terminates in a safe set. The operative question is not whether the system can halt now, but whether it remains inside the region from which halting safely is still reachable.
This is a reachability problem, and it maps onto latent safety filters, which apply Hamilton-Jacobi reachability within the latent space of a generative world model. That formulation matters for contact-rich manipulation, where an analytic dynamics model adequate for computing reachable sets does not exist. Uncertainty-aware extensions treat calibrated epistemic uncertainty as an out-of-distribution proxy, bounding the region where the learned dynamics, and therefore the reachability computation built on top of them, retain validity.
Outside constrained settings this remains unsolved, and the gap is worth stating directly. A system that detects hazard after leaving the recoverable set has produced a measurement, not a mitigation.
2. Uncertainty-triggered compute allocation
The second reading is tractable and mostly an engineering problem: widen the search, sample additional rollouts, increase denoising steps, but only when the state warrants it. Uniform deliberation wastes budget on the forty-ninth routine extraction.
The design question is what triggers the additional allocation. Prediction is the wrong trigger. A model confidently predicting failure and a model with no basis for prediction call for different responses, and only the second indicates that further computation will change the answer. Confident predictions do not improve with more sampling; they are already at the model's resolution limit.
Hardware imposes a useful constraint here. On an Orange Pi Zero 2W the per-action latency budget forces "allocate more compute" to mean a bounded, measured increase rather than an open-ended search, which in turn forces an explicit statement of what the additional compute purchases.
3. Calibrated deferral
The third reading is escalation to a human operator. Implementing it badly takes an afternoon: threshold the model's confidence and raise an alert. Implementing it correctly is harder, because the requirement is not that the system asks when uncertain. The requirement is a deferral rate that can be stated before deployment.
Conformal prediction supplies this. Rather than relying on raw model confidence, which is typically miscalibrated in exactly the distributional regions where deferral matters, calibration against held-out data yields a distribution-free guarantee: a target task success rate subject to a bounded intervention rate. Work in this direction, including KnowNo, converts the escalation decision from a tuned heuristic into a statistical guarantee.
An operator can staff around "this system requests input approximately once per forty actions." No staffing model accommodates "this system requests input when its confidence is low."
Failure modes and composition
| Mechanism | Question answered | Failure mode |
|---|---|---|
| Abort reachability | Is a safe stop still reachable? | Silent exit from the recoverable set |
| Compute allocation | Is more inference warranted here? | Uniform spend, or spend applied too late |
| Calibrated deferral | Should a human decide this? | Unpredictable intervention rate |
The three compose in one direction only. Deferral has no value if the operator is consulted after the recoverable set has been exited, and additional inference compute has no value if it is spent deliberating past the last timestep at which an abort was available. Reachability is the substrate. The other two are scheduling policies that assume it holds.
Most of the runtime-monitoring literature sits orthogonal to all three. Approaches that verify task progress with a vision-language model, or estimate failure likelihood from a policy's internal features, produce detectors. A detector is not a controller, and detecting that the system is in trouble is a separate achievement from having a control sequence that leads out of it.
Scope of the Jenga testbed
The safety-gate experiment addresses the first two mechanisms in a constrained setting. A block extraction is a discrete commitment with a physically identifiable point of no return, which makes "was an abort still available" an empirical question rather than a definitional one. The gate implements the crudest version of the second mechanism: a locked threshold on an anomaly score producing a binary veto, with no graduated response.
Starting at that level is deliberate. If a binary veto on an uncertainty signal fails to reduce catastrophic failures in a domain this constrained, the result bounds what more elaborate mechanisms can be expected to deliver, and it is cheaper to obtain that bound now than after building the elaborate version on top of an unverified assumption.