Uncertainty, Not Prediction, Should Hold the Veto

The usual way to gate a learned manipulation policy is to ask the model what it predicts will happen and act on that prediction. This inverts the reliability ordering. A world model's accuracy degrades fastest on states it has seen least often, which is precisely the regime a safety mechanism exists to handle. Its point prediction is the weakest available signal at the moment you depend on it most.

Epistemic uncertainty behaves differently. It measures distance from the training distribution rather than asserting an outcome, so it degrades gracefully instead of confidently. The thesis under test is therefore narrow and falsifiable: a learned world model's epistemic uncertainty, not its point prediction, should hold veto authority over an action.

Jenga extraction on an SO-101 arm is a reasonable testbed for this. The state space is low-dimensional and physically legible, ground truth is cheap to obtain, and failure is binary. The tower stands or it does not.

Starting conditions

VR teleoperation is functional. Nothing else is. Each remaining component produces an artifact the next one consumes, so the build order is fixed.

Compute placement

The first decision is which machine runs which component, because it determines the dependency set installed at every later step. Getting it wrong surfaces six steps downstream, after the wrong framework is already embedded.

Only the distilled student model executes on the Pi. The V-JEPA teacher is an offline expense paid once, which is the reason for distilling at all rather than deploying the teacher directly.

Ground truth cannot be learned

Steps 1 and 2 are conventional computer vision. Two rigidly mounted cameras, fixed lighting, ArUco markers at the tower base establishing a world frame, edge detection for per-layer block occupancy, and a lean estimate from the side camera.

Nothing in that pipeline is learned, and that constraint is load-bearing. If tower state came from a learned model, then evaluating a learned anomaly detector against it would be circular. Agreement between two models trained on overlapping data is not evidence that either is correct. Classical CV yields a reference signal whose failure modes are uncorrelated with the systems it is used to score.

The exit condition is 20 correct reconstructions out of 20 manual validation trials. Anything below that invalidates every metric computed downstream, so the pipeline stops there until it passes.

Data collection

Twenty-two sessions minimum, allocated across regimes rather than collected until the dataset feels adequate.

The three recovery episodes carry disproportionate weight. A dataset composed entirely of clean successes contains no demonstration of corrective action, so a policy trained on it has no behavior to fall back on when a tower starts to shift.

Time alignment between frames, joint states, and actions is verified per episode rather than assumed. Misalignment in teleoperated data is silent, survives visual inspection of individual frames, and corrupts every model trained afterward.

Control condition and treatment condition

Step 5 trains an ACT policy and runs it autonomously with no gate across held-out scenarios. Those numbers are frozen on recording. A control condition revised after the treatment results are known is not a control condition.

Step 6 builds the detector in two parallel tracks. Track A runs on the GPU machine: pull the smallest published V-JEPA checkpoint, extract embeddings over the Step 4 frames, and train a shallow student network to regress an anomaly score from raw pixels against the teacher's embedding-space prediction error. Track B is manual labeling of roughly 75 held-out frames as structurally normal or risky, reserved for evaluation and excluded from training.

Step 7 wires the gate as an Invariant DAG node taking a proposed action and an anomaly score, emitting execute or block, and then locks the threshold. Tuning happens against the Step 6 validation set only. The constraint is enforced in the API rather than by convention: the gate raises on evaluation with no locked threshold, and raises again on any attempt to re-lock. Changing the threshold requires constructing a new gate, which makes the decision explicit in the code and in the commit history.

The gate fails closed. Any exception raised inside the scorer is caught, recorded as a detector failure, and resolved as a block with an infinite anomaly score. A gate that fails open when its own scoring path breaks provides no guarantee in the one condition where a guarantee matters.

The comparison

Step 8 reruns the identical policy across the identical held-out scenarios with the locked gate in the loop. Step 9 subtracts.

Three conditions have to hold jointly for the thesis to survive. Catastrophic failure rate drops by a margin above noise, success rate does not collapse, and the gate's false-positive rate stays low enough to establish that it discriminates rather than refuses. A gate that blocks every action achieves a zero failure rate and has no utility, so the false-positive ceiling is not a secondary metric.

The verdict logic is implemented with explicit numeric thresholds rather than evaluated by inspection, which removes the opportunity to reinterpret a marginal result favorably after seeing it.

Expected outcomes

There is a real possibility that the gate produces no measurable improvement, or improves the failure rate only by acting so conservatively that task throughput drops to near zero. Both results get published as measured. A negative result with sound methodology constrains the design space; a positive result assembled by adjusting criteria after the fact does not, and in safety work it actively misleads whoever builds on it next.

The build is sequential and currently in progress. The repository carries the plan as its specification, a progress tracker updated as steps close, and an append-only event log so that a failure can be attributed to a specific node rather than reconstructed from video afterward.