An intervention provides only partial evidence about this principle. Before takeover, the robot executes the VLA actions; afterward, the human provides a preferred continuation. The takeover time localizes the context that prompted the intervention and generalizes to other similar situations, while the subsequent human-controlled segment provides a preferred continuation from the reached state toward a more optimal situation. A description such as “keep the mug upright when placing it on the plate” identifies the semantic property motivating the takeover, which may remain valid despite changes in object poses, robot configuration, background, or even task. Trajectories alone leave ambiguous which property mattered; language alone lacks the sensorimotor grounding needed for visuomotor control. Together, they constrain a reusable correction.
We evaluate OopsieRecover on three tasks in simulation and the real world using pi0 in RoboCasa and DROID, with only 5–8 corrections per task. OopsieRecover improves real-world task progress from 35.1% to 61.0%, a 74% relative improvement over the next-best baseline, and transfers corrections to unseen poses, objects, and scenes. Its gated costs generalize zero-shot to novel variations in object poses and scenes, and successfully steer the base policy. Our contributions are: a semantic, abductive formulation of few-shot VLA correction from human-in-the-loop interventions; a generate-and-test procedure that synthesizes and verifies gated costs; and a test-time steering mechanism that applies and generalizes corrections without retraining the VLA.
Collecting Semantic Corrections
OopsieRecover synthesizes a unifying correction objective from each type-based group of corrective interventions. To do so, it first defines a representation capable of capturing transferable intent.
Corrective Program Synthesis and Refinement
The first step is to identify the entities involved in the correction and ground them with geometry in the recorded observations. For PlaceMug, these may include the mug, the obstructing machine region, and the target tray. The task instruction and correction description are used to localize these entities, then construct and track their 3D point-set representations.
Conclusion
OopsieRecover is a framework that learns transferable corrections for frozen VLAs from a handful of semantically annotated human interventions. It uses abduction to synthesize context-gated costs over semantically grounded 3D point sets. These costs determine when steering is needed and rank candidate actions by their predicted consequences. Experiments in simulation and the real world show improved execution and transfer across object-pose and scene variations without additional corrective data. Compared to HG-DAgger, OopsieRecover achieves higher success with fewer interventions, supporting reusable corrective intent as a data-efficient alternative to directly imitating corrective actions.
Acknowledgments
This work was conducted in the Robot Interactive Intelligence (RobIn) Lab at The University of Texas at Austin and was supported in part by TRI (R-YFR-04-Martin-Martin), the National Science Foundation (CAREER FAIN 2543745), the NVIDIA Academic Grant Program, Amazon, and the DARPA TIAMAT program (HR0011-24-9-0428). The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government or any sponsor.
Frequently Asked Questions
What does OopsieRecover learn from a human intervention? It learns a reusable correction objective from the intervention context, the semantic property motivating the takeover, and the human-controlled continuation.
How does OopsieRecover represent entities involved in a correction? It grounds entities in recorded observations using geometry, constructing and tracking their 3D point-set representations.
Does OopsieRecover require retraining the VLA? No. Its test-time steering mechanism applies corrections without retraining the VLA.
How many corrections were used per task in the evaluation? The evaluation used only 5–8 corrections per task across three tasks in simulation and the real world.