Current state-of-the-art Vision-Language-Action models (VLAs) suffer from a fundamental bottleneck: their generalization is strictly bounded by the scale of expert human demonstrations, constrained by the prohibitive cost of scaling such demonstrations. This work challenges the brute-force scaling regime by pretraining manipulation priors using cheap, task-agnostic data, thereby significantly reducing the dependency on expensive expert demonstrations.
The Data Wall in Robot Learning
For years, visuomotor control was dominated by task-specific policies trained within constrained environments. While effective for isolated skills, these methods struggled to generalize to novel objects or unstructured language instructions. Despite their impressive performance, current state-of-the-art VLAs still suffer from a fundamental "data wall" - generalization is strictly bounded by the scale of expert human demonstrations.
Dynamics Learning in Robotics
Most prior methods treat dynamics learning either as an auxiliary objective or a tool for pseudo-labeling data. In contrast, this work employs inverse dynamics as a standalone pretraining phase specifically to unlock the value of massive, task-agnostic action data. By learning physical priors such as object affordances and kinematics before encountering any task semantics, this method provides a robust structural foundation that significantly enhances downstream learning efficiency and performance.
Stage 2: Task-Specific Alignment
Input Representation. The input structure shifts from visual-goal conditioning to language-instruction conditioning. A text encoder maps a language instruction to a sequence of tokens, and the model now receives the combined visual and linguistic input.
Training Objective. The model is finetuned via standard behavior cloning.
Experiments
The experiments are designed to verify whether self-supervised physical priors can effectively bypass the expert data bottleneck, structured around three hypothesis-driven questions:
RQ1 (Effectiveness & Efficiency): Can task-agnostic interaction data, combined with inverse dynamics pretraining, match or exceed the performance of models trained on massive expert datasets while using significantly less labeled data?
RQ2 (Mechanism): Does task-agnostic pretraining improve low-level physical affordances (e.g., grasping, contact), as evidenced by sub-goal success rates and learned visual representations?
RQ3 (Robustness): Does pretraining on diverse, autonomous exploration data improve resilience to real-world distribution shifts, including visual perturbations and environmental clutter?
Experimental Setup
To rigorously evaluate the Decomposition Hypothesis, TAP is benchmarked across both simulated (SIMPLER) and real-world (WidowX 250) environments.
Model & Baselines. The framework uses a Qwen2.5-VL (3B) backbone coupled with a SigLIP visual encoder. TAP is compared against two categories of baselines:
(1) Standard BC: An identical architecture trained from scratch purely on limited expert data, serving as the primary experimental comparison to isolate and prove the value of task-agnostic pretraining.
(2) Nora: A state-of-the-art large-scale VLA baseline trained on massive expert datasets, representing the upper bound of pure expert-supervised learning.
Evaluation Protocols. In simulation, four distinct manipulation tasks are evaluated, with success rates averaged over 50 episodes per checkpoint. In the real world, over 600 physical trials are conducted across five testing conditions, from in-domain setups to severe out-of-distribution scenarios.
Error Analysis
To provide a comprehensive understanding of the TAP method, failure cases from real-world WidowX experiments are systematically analyzed and categorized into two primary modes:
Execution and Dynamics Failures (Approximately 25% of failures): These errors occur when the policy correctly identifies the target object and attempts the right sub-task, but fails during fine-grained physical contact. Common manifestations include end-effector slipping off the object, millimetric pre-grasp misalignment, or depth ambiguity caused by singular camera viewpoints. Task-agnostic pretraining significantly mitigates these issues compared to standard BC, but purely reactive VLA models still struggle with complex, out-of-distribution 3D spatial reasoning under extreme visual shifts.
Semantic and Reasoning Failures (Approximately 75% of failures): These errors are characterized by flawless physical execution directed at the wrong semantic goal. For instance, in the presence of visual distractors, the robot might execute a perfectly smooth grasp on a distractor object rather than the target instruction. In longer horizon sequences, the model occasionally experiences "freezing" or repetitive looping, losing track of the overarching linguistic instruction.
Conclusion
This work introduces Task-Agnostic Pretraining (TAP), a two-stage framework that decouples the learning of physical affordances from semantic task understanding in Vision-Language-Action models. The key insight - the Decomposition Hypothesis - posits that "how to move" can be learned entirely from cheap, unlabeled interaction data, reserving expensive expert demonstrations for teaching "what to do."
These results challenge the prevailing assumption that scaling expert data is the only path to capable embodied agents. Instead, active, task-agnostic interaction, similar to infant-like motor babbling, provides a complementary and cost-effective foundation for robot learning.
Details of Autonomous Random Play Data Collection
To ensure that autonomous exploration yields safe, contact-rich physical interactions rather than redundant free-space motions, a two-phase constrained procedural generation framework is implemented.
Model Architecture
The framework uses Qwen2.5-VL (3B parameters) as the VLM backbone. The visual encoder is a ViT-based SigLIP (400M parameters) processing input images at 224x224 resolution and producing a sequence of visual tokens. The action head is a lightweight 2-layer MLP projecting the VLM's last hidden state to the 7-dimensional action space. During Stage 1, the visual encoder is frozen and only the VLM backbone and action head are trained; in Stage 2, all parameters are jointly finetuned.
Evaluation Details
For SIMPLER evaluation, the standard protocol is followed, reporting success rates averaged over 50 episodes per checkpoint. All comparing models are trained with the same amount of Bridge data with identical training settings.
Data Efficiency and Computational Cost Analysis
A core motivation of this work is to democratize generalist robot learning by reducing the dependency on massive, curated expert datasets and prohibiting computational budgets.
Data Comparison: Expert vs. Task-Agnostic
Standard VLA models (e.g., OpenVLA, Octo, NORA) rely heavily on the Open X-Embodiment (OXE) dataset, which aggregates over 2 million expert trajectories across varying embodiments. While effective, curating and standardizing such datasets requires immense human effort.
In contrast, the TAP framework minimizes the reliance on expert data:
- Stage 1 (Pretraining): Utilizes purely autonomous, task-agnostic interaction data (e.g., random exploration or play), which is "free" in terms of human labeling cost.
- Stage 2 (Finetuning): Achieves competitive performance using only a fraction of the expert demonstrations (e.g., 200 trajectories in real-world experiments) compared to the millions seen by baselines.
Quantitatively, this approach reduces the demand for expert data by several orders of magnitude while maintaining comparable manipulation proficiency.
Qualitative Analysis and Case Studies
To better understand the mechanisms driving the quantitative improvements, a qualitative analysis compares the TAP framework against Standard BC and Nora baselines across both simulation and real-world environments.
Simulation: Unlocking Task-Irrelevant Data in SIMPLER
A core claim of the Decomposition Hypothesis is that physical affordances required for manipulation can be extracted from task-agnostic data. In the SIMPLER "Put the carrot on the plate" task, when expert data is scarce, the Standard BC model struggles to ground the linguistic instruction in precise 3D geometry, navigating to the general vicinity but failing to execute fine-grained contact dynamics required for successful grasping. Both the Nora baseline and TAP demonstrate robust physical execution, confirming that "how to move" transfers effectively even when pretrained on discarded, task-irrelevant trajectories via Inverse Dynamics.
Real-World: Robustness to Unseen Background Shifts
In real-world WidowX 250 experiments, the distinction between robust physical grounding and brittle visual matching becomes starkly apparent. In the "Push the pumpkin to the left" task evaluated on an unseen background texture, the Standard BC model struggles with basic execution, extending the arm but failing to make proper, sustained contact with the object. The Nora baseline exhibits a severe visual grounding failure induced by the out-of-distribution background, misjudging precise 3D spatial coordinates and completely missing the object. In contrast, TAP successfully leverages its robust, task-agnostic physical priors to accurately isolate the manipulable object from the novel background, making solid contact and deliberately pushing the pumpkin to the correct side.
Frequently Asked Questions
What makes task-agnostic pretraining different from standard VLA training? Standard VLA training requires massive amounts of expert-labeled demonstrations for every skill. Task-agnostic pretraining first learns general physical affordances from cheap, unlabeled interaction data, then uses a small amount of expert data only for task-specific alignment.
How much expert data does TAP actually save compared to baselines? TAP reduces expert data requirements by several orders of magnitude, achieving competitive performance with as few as 200 expert trajectories in real-world experiments, compared to the millions required by baseline models like OpenVLA and Octo.
Why do semantic failures account for 75% of errors in the error analysis? This suggests that while task-agnostic pretraining dramatically improves low-level physical execution, the implicit reasoning capacity of a singular, reactive VLA model remains a bottleneck for high-level task understanding and long-horizon planning.
Does TAP work better in simulation or real-world conditions? TAP demonstrates strong performance in both settings, but its robustness to distribution shifts is particularly evident in real-world experiments, where it successfully handles unseen background textures and visual perturbations that cause severe failures in baseline models.