This paper introduces a pretraining approach that builds a shared video-action latent space, enabling robots to learn control policies from large-scale video data without requiring robot action labels. A semantic visual-action tokenizer is trained to compress both visual observations and robot actions into a unified discrete latent space, and a causal video-action model is then pretrained on this space to predict future visual latents and the latent actions connecting them.
Video-Action Model Pretraining and Inference
Building on the semantic visual-action tokenizer, the second stage pretrains a causal video-action model on the shared video-action latent space. It is organized into two levels: a high-level planner that decomposes the language goal into subtasks and tracks their progress, and a low-level video-action policy that realizes each subtask by predicting future visual latents and the latent actions connecting them. The policy operates at the granularity of chunks, so the index ranges over chunks rather than individual frames.
While the video-action policy handles continuous control within a single subtask, long-horizon tasks require decomposing a high-level goal into subtasks, tracking which subtask is currently active, and deciding when to transition to the next one. The planner is implemented as a pretrained vision-language model (VLM) that is fine-tuned to produce structured subtask context for the policy. The VLM backbone provides visual-language grounding out of the box, allowing the planner to assess task state from sparse visual observations without learning scene understanding from scratch.
Planner-Policy Decoupling
The planner and the policy run at different frequencies and are decoupled through an asynchronous shared buffer. The planner runs in the background at a lower frequency and writes structured subtask context into the buffer; the policy reads the latest context at each action-chunk boundary as its conditioning signal. This ensures that planner inference latency does not block the policy's execution loop.
Structured Planner Output
The planner's output format must exactly match the conditioning fields that the video-action policy sees during training; any mismatch causes the policy to receive out-of-distribution conditioning at inference time. Under this constraint, the planner output is defined as a structured JSON with four fields:
- done: A completion signal indicating whether the current subtask should continue or the planner should transition to the next one.
- instruction: A compact execution directive for the current or next subtask, e.g., "Pick up the red bottle on the table." This is the primary conditioning field consumed by the policy during training.
- generation_instruction: A richer description of the expected robot motion and object interaction within the subtask, giving the policy a more specific execution cue.
- local_scene_description: An observation-grounded description of the spatial layout and object state, providing local context for the policy.
The done field is used only by the high-level scheduler to decide whether to keep or refresh the current subtask. The low-level policy is conditioned on the three textual fields: instruction provides the compact command, while generation_instruction and local_scene_description provide more detailed action and scene context when available.
Data Recipe
LingBot-VA 2.0 is trained with a data recipe that follows a multi-task curriculum: general text-to-image pretraining, general text-to-video pretraining, and finally video-action adaptation on embodied data.
General Image and Video Data
For the general text-to-image and text-to-video pretraining stages, the image and video corpora curated for LingBot-Video are reused. These web-scale image and video data provide broad appearance and dynamics priors before video-action adaptation.
Robot Data
All robot data are reprocessed with a unified annotation pipeline. Each long trajectory is segmented into atomic action clips, and each clip is assigned a language prompt and a global task instruction using Qwen3.5-397B. This relabeling step repairs missing or overly generic prompts in earlier versions and makes the language supervision more consistent across embodiments and datasets.
Simulation Evaluation
The model is evaluated on RoboTwin, a bimanual manipulation benchmark with clean and domain-randomized settings, and task success rate is reported against representative baselines. Following LingBot-VA, a multi-task training setup is adopted where all models are trained on 2,500 demonstrations collected in clean scenes (50 per task) plus 25,000 demonstrations from heavily randomized scenes (500 per task).
Ablation Study
To isolate the effect of the tokenizer, a 1.3B video-action model is trained from scratch for each variant. The WAN2.2 VAE baseline and the proposed tokenizer are trained with the same number of tokens and used to initialize the same downstream architecture. Identical RoboTwin post-training is then applied and success rate is evaluated on the official Easy and Hard splits.
The tokenizer ablation shows that replacing the reconstruction-oriented WAN2.2 VAE with the proposed tokenizer consistently improves RoboTwin performance across prediction horizons. The gain becomes larger for longer horizons, suggesting that semantic visual-action tokenization preserves state information that is more useful for world-action modeling and downstream control.
Conclusion
Several directions remain open. The planner and the policy are currently trained separately and coupled through a fixed interface; tighter joint training may further improve long-horizon consistency. The latent-action space is learned from passive video, and interactive or reinforcement signals could sharpen it toward control. Finally, scaling the pretraining corpus and the sparse backbone further, and broadening embodiment coverage beyond bimanual manipulation, are natural next steps toward a general-purpose embodied foundation model.
Contributors
Zelin Gao, Wei Wu, Kecheng Zheng, Ruonan Zhang, Han Zhang, Jingmei Zhao, Yongtao Huang, Haitao Wang, Jingjing Wang, Chen Song
Frequently Asked Questions
How does the planner communicate with the policy during inference? The planner runs asynchronously in the background and writes structured JSON context into a shared buffer, which the policy reads at each action-chunk boundary without blocking execution.
What data is used for the three-stage training curriculum? The model uses web-scale image and video data for general pretraining, followed by robot data reprocessed with a unified annotation pipeline using Qwen3.5-397B for language relabeling.
Does the tokenizer ablation show benefits over standard VAE reconstruction? Yes, the semantic visual-action tokenizer consistently outperforms the WAN2.2 VAE baseline on RoboTwin, with larger gains for longer prediction horizons.
What type of benchmark is used for evaluation? Evaluation is performed on RoboTwin, a bimanual manipulation benchmark with clean and domain-randomized settings, measuring task success rate across Easy and Hard splits.