In-Context HumanGen Dataset
HumanGen is a collection of human–robot in-context learning pairs generated with the in-context human video generation pipeline. Each pair contains a generated human video instruction and its corresponding robot trajectory with executable actions.
HumanGen is organized by data source into Pre-train ICL (External), Pre-train ICL (In-house), Simulation ICL, and Real-world ICL. Following the same task-diverse sampling principle used to curate Task-diverse VA data, source robot videos are sampled by task rather than by raw trajectory frequency, so that the ICL data are not dominated by repeated executions of a small set of tasks.
Pre-train ICL (In-house)
To further increase task coverage, robot trajectories are sampled by task from in-house robotic datasets. This subset covers multiple robot embodiments, such as bimanual Franka and Galaxea R1 Pro, and contains 3,522 tasks and 30,247 human–robot ICL pairs.
When the in-context human video generation pipeline is used to produce the corresponding human videos for this subset, the ratio of third-person viewpoints is reduced, as is the frequency of scene, object, and object-placement changes. This produces more visually aligned ICL samples while maintaining task diversity.
Simulation ICL
To support cross-task evaluation in simulation, ICL data are generated for 50 RoboTwin tasks. This subset contains 2,500 ICL samples, with 50 samples for each task.
Among them, 43 tasks are used for post-training, comprising 2,150 training samples, while the remaining 7 tasks are reserved as unseen tasks for zero-shot cross-task evaluation.
Real-world ICL
Real-world ICL contains human–robot pairs collected on the bimanual Franka evaluation embodiment. It includes 252 human–robot ICL pairs across the three real-world task families used in the evaluation:
- 120 pairs from 30 object-to-container placement training task combinations.
- 96 pairs from 16 three-object sequential manipulation training task combinations.
- 36 pairs for two-table-leg insertion.
This subset is used for post-training so that the model can fit the evaluation robot kinematics for cross-task evaluation.
Zero-WAM: In-Context World-Action Modeling
In zero-shot cross-task manipulation, the robot must execute a novel task without fine-tuning. This ability is pursued with a world action model pretrained on large-scale human–robot ICL pairs and task-balanced robotic video-action data.
At deployment, Zero-WAM uses either a language instruction or a human video demonstration as the task specification to drive execution of unseen tasks.
Implementation
Inference Details
The pretrained Zero-WAM supports two inference modes.
In language-only mode, the model conditions on the language instruction without the ICL human video, and the video classifier-free guidance scale is set to 5.
In ICL mode, the model conditions on the ICL human video and disables the language instruction. ICL classifier-free guidance is used with a guidance scale of 5.
For both modes, the inference chunk size is fixed to 2, and the action classifier-free guidance scale is set to 1.0.
Conclusions and Discussions
Discussions
Zero-shot cross-task generalization is essential for deploying general-purpose robotic policies in open-ended real-world environments. Progress toward this goal requires both stronger transferable priors in robotic foundation policies and richer task interfaces that can convey intent across multiple modalities.
Human demonstrations are a natural source of such task specifications, making continued growth in their scale and diversity important. Although the experiments focus primarily on stationary tabletop manipulation, future work should extend this paradigm to more complex, dynamic, and unstructured environments, including mobile manipulation and substantially longer-horizon tasks.
Among the available sources of transferable experience, egocentric human video is particularly promising because it can be collected at far greater scale than robot demonstrations. Its use, however, remains complicated by gaps in embodiment, observation, and action between humans and robots.
The semantically aligned human–robot data introduced in this work may offer a bridge between abundant egocentric human video and comparatively scarce robot trajectories. By establishing correspondence at the level of task semantics rather than requiring exact motion-level alignment, this data formulation could allow robot policies to acquire broad task knowledge from human experience while relying on much less robot data for executable action supervision.
Cross-Task Robotic Manipulation
Cross-task robotic manipulation evaluates whether a policy can execute an unseen manipulation task without collecting robot demonstrations for that task.
Compared with visual generalization, such as changing scenes or object attributes around a seen task, cross-task generalization is more challenging because the model must infer previously unseen task-conditioned dynamics from the instruction and current observation.
World Action Models
To make world action models execute unseen tasks at test time, recent work explores test-time training.
WAM-TTT steers a frozen world action model by updating lightweight memory from raw human videos at deployment. RoboTTT uses fast-weight test-time training to compress long execution histories into the policy state.
These methods enable unseen-task execution at test time, but still require test-time adaptation through memory or fast-weight updates. In contrast, Zero-WAM takes a different path: it builds human–robot ICL data and task-balanced video-action data into pretraining and post-training, so the policy can directly follow in-context human video instructions for zero-shot cross-task generalization at test time.
Human Video Data for Robotic Manipulation
Zero-WAM targets this missing data regime. Instead of collecting human demonstrations task by task, HumanGen data are automatically generated from task-sampled robotic video-action data.
Because each generated human video is paired with a robot trajectory that retains executable actions, HumanGen scales human video instructions together with task-diverse robot dynamics. This makes human video task specification a scalable component of world action model training for zero-shot cross-task generalization.
Frequently Asked Questions
What is Zero-WAM?
Zero-WAM is a world action model that uses language instructions or human video demonstrations to execute unseen robotic tasks without fine-tuning.
What is included in the HumanGen dataset?
HumanGen contains generated human video instructions paired with corresponding robot trajectories that retain executable actions.
Why is task-balanced robotic video-action data used?
Task-based sampling prevents the in-context learning data from being dominated by repeated executions of a small number of tasks.
What are the main limitations and future directions?
The experiments focus primarily on stationary tabletop manipulation, while future work should address dynamic and unstructured environments, mobile manipulation, and substantially longer-horizon tasks.