AutoIntervene is an online intervention framework for improving the deployment reliability of action-chunking policies. It evaluates each proposed action chunk against successful visual-action support and transfers control when the proposal repeatedly becomes unsupported. Phase-local references determine when control should pass from the policy to the operator, while global references determine when it should return to the policy, allowing autonomy to resume from any supported task phase after operator correction. At each adaptation round, separate switching thresholds are calibrated offline by recomputing proposals and support scores on held-out successful expert demonstrations under the corresponding retrieval scope.
The contributions of this work are threefold:
- A bidirectional intervention framework for action-chunking imitation policies, using phase-local support for policy-to-operator transfer and global support for the return to policy control.
- A mode-specific visual-action calibration procedure that uses held-out successful expert demonstrations to recompute separate intervention and recovery thresholds at each adaptation round under their respective retrieval scopes.
- An evaluation of AutoIntervene on nine real-world tasks, showing that targeted intervention trajectories support iterative policy improvement with substantially less operator-control time than collecting additional full demonstrations.
Related Work
Interactive Imitation Learning
Interactive policy adaptation traditionally relies on an operator to monitor policy execution and manually decide when to take over and return control. Robot-gated methods instead request intervention automatically from deployment-time signals. LazyDAgger uses policy–expert action discrepancy to trigger expert involvement, whereas RND-DAgger uses state novelty estimated by random network distillation.
AutoIntervene follows this robot-gated direction but treats control transfer as a bidirectional, support-based process: phase-local support governs policy-to-operator transfer, global support governs operator-to-policy return, and retained recovery segments provide corrective data for subsequent policy adaptation.
AutoIntervene
Control Setup
The intervention loop is implemented using ALOHA-style leader–follower teleoperation with TriPilot-FF-style arm-side force reflection. The leader arms serve as the operator input, while the follower arms interact with the task.
Under policy control, the leader and follower arms receive the same policy-generated joint-and-gripper commands and remain aligned. When control passes to the operator, each follower arm switches to tracking the corresponding operator-manipulated leader arm, allowing correction to begin without repositioning the leader arms.
Visual-Action Query Construction
For the bimanual system, the commands are grouped by arm:
- One group contains the left-arm joint-and-gripper commands.
- The other contains the corresponding right-arm commands.
Together, the visual embeddings and predicted action prefix form the visual-action query.
The visual-action query combines visual representations with the predicted sequence of upcoming robot actions.
Targeted Intervention Improves Policy Adaptation
Across the seven-task benchmark, targeted intervention consistently improves policy adaptation. After two rounds, both Human and AutoIntervene outperform the Initial policy and Additional Full Data in aggregate.
AutoIntervene delivers the strongest improvement, increasing the across-task mean success rate by 49.1 percentage points, with gains across all seven tasks. These results support AutoIntervene as an effective approach for collecting targeted corrective supervision from learner-induced failure states.
Automatic Switching Enables More Efficient Policy Adaptation
Two policy–operator–policy cycles within a single Peg Disassembly trajectory illustrate that AutoIntervene can collect multiple intervention segments from one rollout.
Adaptation Across ACT, Diffusion, and Flow-Matching Heads
To test whether AutoIntervene transfers beyond ACT, the ACT action head is replaced with Diffusion Policy and Flow Matching heads while keeping the shared backbone, robot interfaces, and monitor unchanged.
All three action heads improve over successive adaptation rounds, and their R3 policies outperform both the Initial policies and Additional Full Data. AutoIntervene therefore transfers across different action-generation mechanisms without head-specific modification.
Conclusion
AutoIntervene combines phase-local visual-action monitoring, bidirectional handoff, and intervention-data aggregation. Across nine real-world bimanual tasks, it improved policies over successive rounds with less additional control data than full demonstrations, achieved higher mean success than manual switching, and required less operator-control time on average.
Together with separately calibrated criteria for both switching directions, these components focus operator control on unsupported periods and resume autonomy once the policy proposal returns to demonstrated support. Future work will extend online calibration across broader tasks, perturbations, and operators.
Frequently Asked Questions
What is AutoIntervene?
AutoIntervene is an online intervention framework for improving the deployment reliability of action-chunking imitation policies.
How does AutoIntervene decide when to transfer control to the operator?
It uses phase-local visual-action support and transfers control when the proposed action repeatedly becomes unsupported.
How does control return to the policy?
Global support determines when the operator-to-policy handoff occurs, allowing autonomy to resume from a supported task phase after correction.
Which action-generation mechanisms does AutoIntervene support?
It transfers across ACT, Diffusion Policy, and Flow Matching heads without head-specific modification.