The curation process turns four experience sources into training-ready video–language examples. It includes quality checks, standardized video records, segmentation of long videos into subtasks where applicable, and multi-level captions before the resulting data is mixed for pretraining.
Language–Video Alignment
General Videos
We use OpenVid and Panda70M, totaling 10,909.6 hours. These open-source video-generation datasets contain videos and text descriptions covering diverse scenes, objects, and visual dynamics. We retain their videos and original descriptions to provide language–video experience beyond manipulation demonstrations.
Egocentric Videos
We use Ego4D, EgoVerse, and EgoSuite, totaling 371,212 trajectories and 10,986.3 hours. Ego4D primarily records daily activities across real-life settings. EgoVerse covers more diverse scenes, adding egocentric visual experience across environments. EgoSuite is currently the largest open-source egocentric manipulation video dataset. We retain the videos and language descriptions from these sources.
Data Preparation
We package all sources in LeRobot 2.1 format. We retain only samples whose videos, language descriptions, and metadata can be linked. We exclude samples with unassociated video, language, or metadata; videos with encoding or decoding failures; and known low-quality samples. Video records retain their corresponding language descriptions and metadata. The inventory counts the recorded episodes and videos at their source durations.
Visual Content Filtering
For general video datasets such as OpenVid and Panda70M, we use original annotations and VLM-based content review to remove animations, static scenes, television programs, and other unsuitable content. For egocentric videos, we use a VLM to assess motion blur, brightness, occlusion, and whether a clip contains meaningful actions, reducing the impact of low-quality imagery and uninformative behavior on training. The UMI and robot demonstrations described below use similar filtering rules.
Video Model Adaptation on General Videos
Our WAM modifies Wan’s video modeling design to support robot control, which may disrupt pretrained visual and dynamics priors. To ease this adaptation, we first train the video branch on general videos, whose distribution is close to that of Wan’s pretraining data. During this stage, we use the same input frame count, spatial resolution, and causal attention scheme as subsequent WAM training. This stage aims to preserve these priors during adaptation, before introducing embodied data.
Language–Video–Action Alignment
UMI and robot demonstrations follow the video curation and filtering procedures described above and use the language annotations described in the preceding section. Building on these language–video correspondences, we align video segments with state and action records to construct language–video–action supervision. We describe the collection and curation of these demonstrations below.
During collection, head and bilateral wrist views record the overall manipulation scene and local interaction details, respectively. Smartphone visual–inertial tracking provides action trajectories, while external encoders record gripper opening and closing. A shared coordinate system and time reference establish spatial and temporal correspondences among the videos of both hands, motion trajectories, and gripper states.
The collection procedure includes equipment checks and synchronization verification before recording, tracking stability and kinematic feasibility checks during recording, and video and AR trajectory playback afterward. These checks identify anomalies such as localization drift and trajectory jumps and verify task completion.
State–Action Representation
UMI and robot demonstrations use a shared 16-dimensional bimanual end-effector representation. Each hand has a three-dimensional position, a four-dimensional rotation quaternion in wxyz order, and a one-dimensional gripper value. State and action records are associated with their corresponding video segments. Each source is normalized using its own q1 and q99 statistics.
Trajectory Quality
For our in-house UMI demonstrations, we construct action-supervision masks from the state and action validity annotations provided by the collection pipeline. Videos participate in video learning, while valid action labels additionally provide action supervision and invalid labels are masked.
For the collected robot demonstrations, we first check action dimensions, missing values, NaNs, non-finite values, and quaternion validity. For subsets with the required joint and end-effector state fields, we further exclude trajectories with abnormal joint angles, excessive changes in joint angles or end-effector poses between adjacent frames, or no movement throughout the trajectory. Trajectories that pass these quality checks are used for action training.
Experiments
We aim to use multisource experience to preserve the language–video capabilities acquired during video pretraining and extend them to generalizable robot control. Our experiments examine how language–video alignment, language–video–action alignment, joint training, and pretraining scale affect downstream generalization. We organize our study around the following four questions:
- How can effective cross-modal alignment be established?
- How can multisource experience be integrated?
- Does scaling pretraining improve control generalization?
- How well does our model perform in unseen environments?
Experimental Setup
We pretrain the model using the architecture and multisource corpus introduced above. The video branch is initialized from Wan2.2-TI2V-5B, and the action branch is randomly initialized. All model parameters are updated during training. Pretraining and post-training both use the joint video and action objective defined above. We use AdamW with an initial learning rate whose value is represented by a formula in the source text and a default global batch size of 256. Training steps and specific settings for each study are given in the corresponding experiment descriptions and table captions.
Discussion and Conclusion
We studied how to extend pretrained language–video capabilities to generalizable robot control with multisource experience. Using approximately 47,000 hours of experience, we systematically investigated language–video alignment, language–video–action alignment, and their joint learning.
We found that detailed and accurate language descriptions preserve pretrained language–video capabilities. Meanwhile, action supervision from UMI demonstrations further supports robot control, demonstrating the value of human manipulation experience for robot learning. Our findings show the importance of experience organization and supervision usage. The resulting training recipe demonstrates generalization to unseen conditions in simulation and real-world evaluations, with further gains from continued pretraining. We release the data, code, and model weights to support future research.
Built upon the pretrained video model, the proposed method offers limited support for complex reasoning and explicit multi-stage planning, despite outperforming VLA models in our control generalization evaluations. The limited historical context may also hinder the execution of long-range tasks. In the future, we plan to explore memory mechanisms and hierarchical planning to support tasks that require reasoning over past observations and planning subsequent actions.
Acknowledgments
We thank Zijun Xu for his contributions to the early exploration of this work. We also thank Yutong Dong for her efforts in real-world deployment.
Common Settings
Pretraining and post-training share the optimization settings specified in the paper’s referenced table. Batch sizes, training steps, and learning-rate schedules vary across training stages, as detailed in the referenced training-configurations section.
Training Configurations
In pretraining, we initialize the video DiT from Wan2.2-TI2V-5B and load its pretrained UMT5-XXL text encoder and Wan2.2 video VAE. The action branch, including the action and state encoders, transformer blocks, and action decoder, is randomly initialized. For each experiment, post-training starts from the corresponding pretrained weights.
Post-Training Data Preparation
We adapt models separately to cup stacking, towel folding, and gift-in-box using task-specific data for post-training following general-purpose pretraining on egocentric videos and UMI demonstrations. The paper’s referenced table summarizes the post-training data for each task.
Evaluation Protocols and Metrics
We evaluate under Easy (C2C) and Hard (C2R) settings. Easy settings retain the scene conditions and physical objects used in post-training, including their shapes and colors, while varying their initial configurations across layouts. Hard settings vary across scene and object attributes, including background, tabletop material, object instance, shape, color, size, and texture.
All models use synchronous inference and execute actions at 30 Hz.
Real-World Task Success Rates
The paper reports SR and PSR for each real-world task and their average under the Easy and Hard settings.
Frequently Asked Questions
What experience sources are used for language–video pretraining? The paper uses general videos from OpenVid and Panda70M, and egocentric videos from Ego4D, EgoVerse, and EgoSuite.
How are robot actions represented? Each hand uses a 16-dimensional bimanual end-effector representation: position, a wxyz rotation quaternion, and a gripper value.
How does the paper filter demonstration trajectories? It checks action dimensions, missing and non-finite values, quaternion validity, and—where applicable—joint angles, pose changes, and movement.
What are the stated limitations and future directions? The method has limited support for complex reasoning, explicit multi-stage planning, and long-range tasks. The authors plan to explore memory mechanisms and hierarchical planning.