Progress Probes vs. VLMs as Runtime Monitors
We study how our lightweight progress probe compares to general-purpose VLMs as runtime monitors.
To calibrate the Progress head, we run Robometer on the same training trajectories used for Progress Probe and aggregate its predicted progress curves for each task, defining nominal progress from Robometer’s own predictions on successful trajectories. We evaluate two variants: Robometer-ZS (zero-shot) and Robometer-FT, fine-tuned on the same success-and-failure dataset used to train SAFE.
Discriminability by Failure Detection
We test whether Progress Probes scale to monitoring a real-world VLA policy, evaluating how well they distinguish successful from failed execution when deployed on the real robot.
Method. We evaluate each baseline on the hardware evaluation dataset and ask the monitor to determine (a) whether the entire episode is a successful or failed rollout via max aggregation and (b) given a trajectory, whether the current time-step is a successful state or failure state.
Conclusion and Limitations
We introduced Progress Probes, a lightweight approach to runtime monitoring that linearly decodes task progress from a VLA’s internal activations. Across simulation and hardware experiments, Progress Probes achieve competitive failure-monitoring performance while being substantially more computationally efficient than existing VLM-based monitors. These results suggest that task progress encoded within VLA activations provides a simple yet effective signal for runtime failure monitoring.
Frequently Asked Questions
What do Progress Probes monitor? They decode task progress from a VLA’s internal activations to support runtime failure monitoring.
How is Robometer used in the evaluation? Robometer’s predicted progress curves on successful trajectories define nominal progress. The study evaluates zero-shot and fine-tuned Robometer variants.
What failure-detection settings are evaluated on the real robot? The monitor classifies entire episodes using max aggregation and classifies current time-steps as successful or failure states.
How do Progress Probes compare computationally with VLM-based monitors? They achieve competitive failure-monitoring performance while being substantially more computationally efficient than existing VLM-based monitors.