Related Work
Existing methods can detect failures, but their alarms are not necessarily timely, informative, or sufficiently calibrated. VLA-Tug addresses these limitations through failure-aware representations, onset-based detection, and joint statistical guarantees.
Problem Statement
This section gives broad definitions used to describe how failure detectors for vision-language-action models (VLAs) function, formulates the problem mathematically, and introduces how VLA-Tug addresses it.
Definitions
To facilitate comparisons across methods, we define the concepts of failure score, detection threshold, detection time, failure time, and failure mode.
- Failure score. The score, a probe’s failure-belief value at each timestep, encodes the probe’s belief that the VLA is failing. It need not be a probability in the strict sense.
- Detection threshold. The threshold, the cutoff applied to failure scores, separates failures from successes.
- Detection time. The time, the first timestep at which the score crosses the threshold, is the first time the score crosses the threshold.
- Failure time. The time, also called the failure onset, is when the policy makes an obvious mistake or its progress stalls to the point where an intervention, human or otherwise, is needed.
- Failure mode. The label, a semantic category for the failure, is a high-level category assigned to each failure, such as grasping failure, placement failure, or no progress.
Methodology
VLA-Tug consists of a failure probe and a calibrated alarm threshold. The following sections describe the probe and its training, then explain how the alarm threshold is calibrated and what guarantees this provides.
Failure Detection Through Latent Space Probing
Building on existing work, we instantiate the probe as a combination of LSTMs and MLPs. The goal is to capture a nonlinear pattern in the backbone embeddings that generalizes beyond the episodes seen during training. We introduce the model inputs, describe the model architecture, and explain the training strategy.
Evaluation of Informativeness
This section focuses on failure-mode prediction, a crucial element for enabling failure prevention through steering. Failure modes are strongly imbalanced, so we report balanced accuracy: the mean, over classes, of the share of their failures predicted correctly. Since existing baselines do not predict failure modes, we compare against a majority baseline and focus on LIBERO-X.
The practical consequence is that the probe narrows a failure down to three candidate classes, which are more likely than the majority baseline to include the true one. Different failure modes carry different risks, so a practitioner can use these candidates to select the level of response—for example, responding to the riskiest of the three predicted classes with a stop, retry, or handover to a human—instead of applying the same intervention to every failure.
Backbone embeddings carry enough information to predict the failure class above a majority baseline, letting a practitioner match the response to the predicted risk.
Practical Considerations
Any real-world system must balance performance—timeliness (T), calibration (C), and informativeness (I)—with the cost of building and running it (efficiency, E). We evaluate the costs of labeling the dataset, training the probe, calibrating it, and running it at inference time.
Labeling costs about one cent per failed episode (under $50 for all annotations, including queries not used in this paper). Labels are therefore cheap, and practitioners will often already have unlabeled rollouts from system deployment.
Steering Demonstration
We demonstrate how VLA-Tug’s four properties—timeliness, calibration, informativeness, and efficiency—can jointly support failure prevention through VLA steering. The goal is to illustrate how an effective failure predictor can provide signals to guide interventions, rather than to develop a steering method itself.
We consider a steering architecture in which an expert model, such as a vision-language model (VLM), can invoke failure-recovery tools in response to the probe’s predictions. For our experiments, we implement these tools using failure-mode-specific heuristics in simulation and demonstrate corresponding interventions on a real robot.
For example, in a simulated rollout, the task is “place the largest bowl inside the left compartment of the bowl drainer.” The probe raises an alarm before the bowl is mistakenly placed in the compartment on the right. After steering the end effector towards the left side, the policy resumes and the rollout succeeds. Without intervention, the bowl would be placed on the right side and the rollout would fail.
Our real-world experiments similarly demonstrate that VLA-Tug’s joint properties help steer a VLA away from failure. In the “put the mushroom on the plate” task, without VLA-Tug the gripper misses the mushroom and hits the table. With VLA-Tug turned on, the probe predicts the imminent grasp failure, and corrective action is taken. In this example, the corrective action is to pause the VLA, let an expert model shift the trajectory to a successful one, and resume the VLA.
Such a strategy is possible with a failure detector like VLA-Tug, which anticipates failure before it happens, rather than after it is too late. The approach is agnostic to the steering architecture. Exploration of robust steering strategies is left for future work. These examples demonstrate the applicability of VLA-Tug in safe and principled robotic systems.
The proof-of-concept demonstrations show how VLA-Tug can provide the timing and failure-mode information needed for targeted interventions during VLA execution. Timeliness creates an opportunity to intervene, calibration controls when interventions are triggered, informativeness guides which intervention is applied, and efficiency allows the probe to operate alongside the VLA.
Conclusion
First, the calibration guarantees rely on exchangeability between the calibration and test episodes, which weakens as distribution shift grows. Empirically, the degradation is gradual. Second, the choice of source backbone embedding is not analyzed systematically. Third, the steering component, in which an expert model intervenes on a predicted failure, is heuristic and is not evaluated in the scope of this work.
Probe and Training
At every policy step, we average the backbone tokens of the agent camera, the wrist camera, and the unpadded language prompt, then concatenate the three means into a single feature vector. No other feature of the backbone is used. All models in this paper, including the baselines, read this vector.
Benchmarks and Splits
We use the 10-suite of LIBERO-Occ, an extension of LIBERO that occludes the agent-view camera. Both policies fail rarely on the original benchmark, so we mix occluded and unoccluded episodes in equal parts to obtain enough failures: 250 of each per policy.
Baselines
We re-implement all baselines on the same inputs and splits as our probe. Where a baseline has a configuration grid, we run the whole grid on every split and select, for every setting, the configuration with the highest mean area on the validation episodes, never looking at the unseen tasks; ties go to the earlier configuration of the grid.
LIBERO-10 and the real robot pool their runs into one choice per method. On LIBERO-X, SAFE uses one configuration for the three kinds of splits, as in its own protocol, and every other method chooses per kind of held-out set. Our probe uses one architecture in every setting.
Metrics
The reported metric is expressed as a percentage and scaled so that a detector with the specified ideal behavior scores 100.
Informativeness Details
This section gives details behind the evaluation of informativeness.
The failure modes are strongly imbalanced, so we report balanced accuracy: the mean over classes of the share of their failures predicted correctly. We merge the two smallest classes repeatedly, once per dataset, until every class holds at least 5% of the failures. The rule uses label frequencies only. The mode head is not retrained: a merged class gets the sum of the probabilities of its modes.
Accuracy is computed on predictions pooled over the splits of a setting, since a split has few failures of a rare class. We exclude LIBERO-10 because it has too few failures for a per-class accuracy: even after merging, its smallest classes hold only 6 to 16 failures, so one failure changes a class’s accuracy by several points.
Cost and Latency
This section gives details behind the evaluation of practical considerations.
Ablations
We ablate the supervision strategy, the auxiliary heads, and the failure time used for evaluation.
Steering Demonstrations
Steering is used only to show how the VLA-Tug signal could be used; the heuristics are simple and are not evaluated. When the score crosses the calibrated threshold, the policy is paused, an intervention chosen from the predicted failure mode is applied, and the policy resumes.
We implemented three heuristics for LIBERO-X. For a wrong-target or placement/insertion failure, the end effector is displaced a small distance towards the destination. For a grasp failure, it is displaced laterally, perpendicular to its recent motion. For every other predicted mode, the policy replans from the current state.
On the task “place the white-yellow mug on top of the shelf,” after the alarm the probe predicts the failure mode, the matching intervention is applied, and the policy resumes and succeeds. Without intervention, the mug is placed on the table and the task fails.
Frequently Asked Questions
What does the VLA-Tug probe use as input?
At each policy step, it concatenates mean backbone tokens from the agent camera, wrist camera, and unpadded language prompt. It uses no other backbone features.
How does VLA-Tug define failure time?
Failure time, or failure onset, is when the policy makes an obvious mistake or progress stalls enough that an intervention is needed.
How does the probe help choose a response to failure?
It narrows a failure to three candidate classes, which can help practitioners choose an intervention appropriate to the predicted risk.
What are the limitations of the steering demonstrations?
The steering heuristics are simple and are not evaluated as robust steering strategies. The calibration guarantees also rely on exchangeability between calibration and test episodes.