The reliable deployment of visuomotor policies under camera perturbation requires knowing the range of actions the policy may take when its input is uncertain. Verifying such policies end-to-end is intractable because propagating a set through a large visual encoder accumulates relaxation error, leaving the resulting action set expensive to compute and too loose to be useful. This work keeps the visual encoder frozen and inserts a calibrated low-dimensional interface between it and the downstream policy, so that camera-pose uncertainty is carried in a few dimensions and set propagation runs only through the small downstream policy, making set-based verification tractable for visuomotor policies.
Introduction
Set-based training makes the size of a propagated output enclosure an explicit training objective, so the trained network produces tighter enclosures that are easier to verify. Applying it end-to-end to a modern visuomotor policy is intractable, since propagating a set through a large visual encoder accumulates relaxation error across many layers and leaves the resulting action set both expensive to compute and too loose to be useful.
The key idea is to keep the visual encoder frozen and insert a calibrated low-dimensional interface between it and the downstream policy, so that the camera-pose uncertainty is carried in a few dimensions and set propagation runs only through the small downstream policy. Set-based training then contracts the propagated output enclosure directly, which is tractable where end-to-end propagation is not. Split conformal calibration of the sampled action deviations then reports a radius with finite-sample coverage.
Contributions
The contributions of this work are the following:
- Decoupling through a calibrated interface: The frozen visual encoder is decoupled from the downstream policy through a calibrated low-dimensional interface, and set-based training is applied to the policy alone, which makes set-based verification tractable for a visuomotor policy.
- Proof of enclosure width as an upper bound: The trained enclosure width is proven to upper-bound the physical action deviation induced by a sampled camera perturbation, which links the training objective to the calibrated radius reported at deployment.
- Empirical reduction on a manipulation benchmark: On a manipulation benchmark under camera-extrinsic perturbation, set-based training reduces the reachable-action radius while preserving closed-loop task capability. Matched behavior-only, observational-consistency, and pointwise-adversarial controls do not reproduce the reduction.
Cross-Task Results
A capacity study across three widths of the flow matching network is reported in the supplement. Direct contraction at the raw 1024-dimensional feature interface attains small training widths but does not pass the behavior criterion, as the supplement reports, which supports the compact bottleneck as the behavior-compatible interface.
Conclusion
The study shows that the geometry of a policy's action set under sensor uncertainty can be shaped during training once perception is decoupled from downstream set reasoning through a calibrated low-dimensional interface, which makes a full-set training signal affordable. The distinction from pointwise adversarial training is mechanistic: the set objective constrains the entire enclosure, whereas a pointwise attack constrains only the point it locates. This mechanism is consistent with the smaller physical action deviations observed for set-based training.
A deterministic map from the pose set to the interface set would upgrade the probabilistic radius toward an end-to-end guarantee.
Frequently Asked Questions
Why is end-to-end set propagation for visuomotor policies intractable?
Propagating a set through a large visual encoder accumulates relaxation error across many layers, leaving the computed action set both expensive to compute and too loose to be useful for verification.
How does the proposed method make set-based verification tractable?
The method keeps the visual encoder frozen and inserts a calibrated low-dimensional interface between the encoder and the downstream policy, so set propagation runs only through the small downstream policy.
What distinguishes set-based training from pointwise adversarial training?
The set objective constrains the entire output enclosure, whereas a pointwise attack constrains only the specific point it locates, which explains the smaller physical action deviations observed with set-based training.
What would upgrade the probabilistic radius to an end-to-end guarantee?
A deterministic map from the pose set to the interface set would upgrade the probabilistic radius toward an end-to-end guarantee.