Quality-aware pretraining helped vision-language-action (VLA) robots learn from imperfect demonstrations, but the best way to use quality labels depended on the model and training stage. Across three VLA policies, approach-aware episode conditioning beat plain behavioral cloning on mean success, and in real-world tests simulation pretraining raised success from 9.6% to 31.7%, with quality conditioning reaching 32.5%.
What Did the Researchers Build?
The researchers built AXIS-Bench, a benchmark for testing how robot policies learn from demonstrations of varying quality. A demonstration can include useful actions alongside hesitation, detours, or other movements that make little progress toward a task. Rather than treating every recorded action as equally valuable, the benchmark derives quality scores from recorded object and robot states, then tests different ways of using those scores during pretraining.
AXIS-Bench is designed to make comparisons controlled: approaches use shared demonstrations and standardized training and evaluation protocols, with matched data and training budgets within each VLA. It compares three strategies. Weighting changes how much each example contributes to training; filtering removes lower-scoring examples; and conditioning gives the policy a quality label as an input. The benchmark also compares labels attached to short action chunks with labels attached to whole episodes, or complete demonstrations.
For real-world validation, the benchmark includes a process for aligning simulated camera views and tabletop scenes with a physical workspace. Recorded actions and object states remain intact while the observations are re-rendered to match the target setup. That gives researchers a way to pretrain on simulation trajectories, fine-tune on real demonstrations, and evaluate the resulting policy on physical tasks.
What Were the Key Results?
The main simulation finding is that approach-aware episode conditioning improved mean task success over plain behavioral cloning across all three evaluated VLA policies. The best setting was not universal, however: conditioning tended to work better with episode-level labels, while filtering tended to favor chunk-level labels. Those preferences could reverse depending on the model and quality definition. Approach-aware chunk weighting also improved mean success over behavioral cloning for the evaluated models, with reported gains of 0.67 and 0.50 percentage points for two of them.
In the real-world study, simulation pretraining lifted success from 9.6% for a fine-tuning-only control to 31.7% for plain behavioral-cloning pretraining. Quality-aware conditioning reached 32.5%. That final increase over plain behavioral cloning was modest, and conditioning improved some tasks while lowering results on others.
| Real-world configuration | Success rate | What it tested |
|---|---|---|
| Fine-tuning only | 9.6% | Real demonstrations without AXIS simulation pretraining |
| Plain behavioral-cloning pretraining | 31.7% | Simulation pretraining followed by plain fine-tuning |
| Approach-aware episode conditioning | 32.5% | Quality-conditioned simulation pretraining and tagged fine-tuning |
The real-world evaluation covered ten manipulation tasks, with 24 trials per task. The plain-pretraining comparison improved success on nine of the ten tasks. These results establish a strong benefit from simulation pretraining in this setup, while showing a smaller additional gain from quality conditioning.
How Does Quality-Aware Pretraining Work?
The method starts with demonstrations that include recorded states over time. The researchers use progress toward the goal to score the quality of a demonstration: examples that move the task forward receive a different score from actions that contribute less progress. The scores can be computed for a short action chunk or for the whole episode. This distinction matters because a brief pause or detour might look unhelpful locally but occur within a demonstration that succeeds overall.
The same underlying quality information can enter training in three ways. With weighting, higher-scoring samples have more influence on the learning loss. With filtering, lower-quality samples are removed; the benchmark’s top-quality filtering keeps half of the preprocessed training time steps. With conditioning, the quality label is supplied to the policy along with its usual inputs, allowing the policy to learn behavior associated with different quality levels.
The experiments also distinguish approach-aware quality from object-centric quality. The results show that the choice of quality definition can change which label granularity works best; the paper does not identify a single combination that wins for every policy. During real-world validation, the researchers pretrained on 19,250 simulation demonstrations, then fine-tuned on 25 real demonstrations for each of ten tasks. Conditioning used episode-score quintiles, with a neutral tag for demonstrations that could not be scored. During target-task fine-tuning, keeping a constant quality tag produced higher mean success than plain fine-tuning in the tested conditioning setups.
Why Does This Matter for Robotics?
Robot learning often depends on demonstrations recorded by people, and real demonstrations are rarely perfectly efficient. A person may hesitate, adjust a grasp, or take a detour. If a training pipeline treats every recorded action as equally useful, those variations can make it harder for a policy to learn what matters. This benchmark tests practical ways to make use of such imperfect data rather than assuming demonstrations are all equally good.
The results also offer a useful caution for teams selecting a training recipe: quality labels are not a plug-in upgrade with one universally best setting. The method, label granularity, base policy, and target-task fine-tuning protocol interact. A team may need to evaluate the full workflow on its own tasks, rather than choosing filtering or conditioning based on a result from another model.
The workspace-alignment process is relevant to teams exploring simulation pretraining for physical manipulation. The researchers report that configuring their reference workspace took about 30 minutes, after which recorded trajectories could be re-rendered for pretraining. That is an encouraging practical starting point, not proof that alignment is equally quick or effective across other robots and workspaces. Teams planning hardware trials can also compare platforms through used robots for sale.
What Are the Limitations and Open Questions?
The benchmark uses a shared tabletop simulation setting, and its physical-world validation covers one reference workspace and one VLA configuration. The real-world test also shows only a small difference between plain behavioral-cloning pretraining and quality-aware conditioning, despite the much larger gain from simulation pretraining over no pretraining. More workspaces, robot embodiments, and larger pretraining budgets are needed to establish how broadly the findings transfer.
Progress-based scores capture only one part of demonstration quality. A useful preparation step or recovery action might make little immediate progress and therefore receive a low score. The study also does not isolate why different base policies respond differently: their initializations and trainable modules vary. Further tests could examine whether conditioning helps because it preserves broader data coverage, and whether preference-based or task-substep quality signals improve the results.
What Do Teams Often Ask About Quality-Aware Pretraining?
Does quality-aware pretraining beat plain behavioral cloning? Approach-aware episode conditioning improved mean success over plain behavioral cloning across the three evaluated VLA policies. The benefit depends on the model and training configuration.
Should quality labels describe action chunks or full demonstrations? Conditioning favored episode labels on average, while filtering favored chunk labels. The preference could reverse for particular models or quality definitions.
Did simulation pretraining help on real tasks? Yes. In this setup, success increased from 9.6% without simulation pretraining to 31.7% with plain behavioral-cloning pretraining.
Does conditioning guarantee better results on every task? No. The real-world conditioned setup reached 32.5% overall, but improved on five tasks, declined on four, and tied on one compared with plain behavioral cloning.
What Should Robot Teams Take Away?
AXIS-Bench shows that imperfect demonstrations can still support effective robot pretraining, and that simulation pretraining substantially improved real-world success in the tested setup. Quality labels add a smaller, configuration-dependent benefit, so teams should compare training choices on their own policies and tasks.