Agentic robot systems provide complementary structure to world action models (WAMs): a programmatic harness selects reusable skills and tools, binds them to the current scene, tracks progress, and replans as observations change. This programmatic representation makes the structure of multi-step tasks explicit, specifying which skills and tools to invoke, with what arguments, and in what sequence. Yet agentic robotics often lacks predictive grounding in real-world physical dynamics and does not leverage a learned model of how the world will change as the robot acts.
We advocate an architecture that combines the foresight and action execution of WAMs with the cognitive efficiency, reusable skills, and potential for future self-improvement of agentic robotics. We propose AW-0, an agentic WAM framework that incorporates code-based planning and generation directly into world action modeling via a grounded code expert. It harnesses broad programming priors to synthesize modular skills featuring explicit parameters and control logic. Code guides synchronized video and control modules within a mixture-of-transformers architecture, simultaneously denoising discrete code structures, predicted visual observations, and low-level motor commands.
Our primary contribution is the conceptualization and realization of agentic robotic skills within a world action model framework. We show that code generation improves action prediction in the world model, and that code generated from the world model can be reused in an agentic framework. Specific contributions include a mixture-of-transformers framework that integrates a code expert into world action models via simultaneous denoising of code, video, and control; an automated agentic pipeline for program annotation and skill discovery; a dataset of structured skill programs; and a hybrid execution policy that dynamically toggles between programmatic skill invocation and direct deployment of action chunks from the WAM action expert.
Mixture-of-Transformer Architecture
Code Expert
The code expert augments a pretrained language backbone with a frozen Qwen3-VL vision transformer and adopts LLaDA-style masked-token diffusion. Current multi-view observations are encoded into visual tokens and prepended to the instruction and program tokens to ground skill planning in the scene.
Video and Action Experts
The video expert uses a video diffusion transformer initialized from Cosmos-Predict2.5 to model VAE-encoded future frames conditioned on observed frames, while a separate action diffusion transformer generates continuous action chunks. Both experts receive language embeddings through cross-attention, with additional robot-state conditioning for the action expert.
Agentic Coding in WAM
Memory Cache
Since a single observation is often insufficient to reveal task progress, the code expert receives a compact cache of the slices generated so far. The history slices are compressed into a set of learnable memory slots and read back as prefix tokens before the prompt in the code stream. During training, the buffer holds the preceding annotated slices, replaced with model predictions at a specified probability. At inference, it is reset per episode and grows with each decoded slice.
Hybrid Actions
We additionally explore an inference mode in which the decoded code slice is executed against a library of perception and control primitives, following the Code-as-Policies approach. The executed actions are integrated with the direct actions of the action expert. At deployment, the two channels are arbitrated at every replanning point: the decoded slice takes over the current stage only if it passes a set of validity and progress checks; otherwise, the action chunk from the same forward pass drives the robot until the next replan.
Experiments
We evaluate AW-0 on RoboTwin 2.0 and eight real-world tasks spanning parallel-gripper and dexterous-hand embodiments. We assess language following, semantic reasoning, and skill execution, as well as the robustness benefits of hybrid execution in variant scenarios.
Conclusion
We present AW-0, an agentic world action model that integrates code-based skill planning with predictive dynamics and continuous control. Through joint denoising, the code expert provides structured semantic guidance for video and action generation. An automated annotation pipeline enables learning this planning interface from large-scale robot and human demonstrations.
Experiments on RoboTwin 2.0 and real-world manipulation with YAM and dexterous hands demonstrate improvements in language following, semantic reasoning, and skill execution, with hybrid execution further enhancing robustness. These results establish a unified foundation for future work on skill reuse and iterative interaction between programmatic planning, world prediction, and execution, providing a potential foundation for agentic systems that continually interact with and learn from environments.
Acknowledgments
Zhuoyang Liu thanks Prof. Shanghang Zhang at Peking University for her mentorship and support.
AI Use Statement
We used Gemini 3.1 Pro and Claude Code with Claude Opus 4.8 in our automatic annotation pipeline to generate demonstration-aligned program labels and distill reusable skill libraries. Claude Opus 4.8 was also used to generate executable programs for collecting simulation demonstrations. Additionally, we used GPT-6 Astra for language polishing and stylistic refinement, as well as part of figure preparation.
Ethics Statement
This work studies code-guided world action models for robotic manipulation using robot and egocentric human demonstrations. All pretraining datasets are used in accordance with their respective licenses and data-use terms. All post-training demonstrations are collected by our team for the tasks evaluated in this work.
Pretraining Dataset Statistics
We pretrain AW-0 on the union of two real-world manipulation datasets: YAM and Egocentric Human data. The YAM corpus contains 73,101 episodes across 279 tasks, totaling 1,919.3 hours of bimanual robot teleoperation. Demonstrations are recorded at 30 Hz with left-wrist, right-wrist, and top-view RGB cameras.
The egocentric human corpus contains 175,541 episodes across 327 tasks, totaling 993.9 hours, with a single head-mounted RGB camera and tracked hand motion retargeted to 22-DoF dexterous-hand joint targets per hand. Together, the two sources comprise 248,642 episodes and 2,913.2 hours. The human camera view is replicated across the three input slots; each training window is paired with its episode-level instruction and aligned program slice.
Code Expert and Language Following
We further examine whether code-based planning improves the alignment of predicted visual dynamics with task semantics. We compare the video generation quality of the right wrist camera for AW-0, a natural-language subgoal variant (Subgoal), and a model without a planning stream (AW-0 without code), under the instruction “Arrange the fruits so that finally there is only mango in the left bowl.” This task requires identifying the intended object and grounding the desired outcome in an appropriate interaction. The three variants are evaluated under the same task and scene configuration, with similar but non-identical initial observations from separate rollouts.
The predicted futures reveal distinct differences in task understanding: AW-0 directs the interaction toward the mango, whereas Subgoal exhibits ambiguous targeting between the apple and mango, and AW-0 without code targets the apple. These observations suggest that structured skill calls and explicit object arguments provide a more specific planning condition for video prediction than a natural-language subgoal alone. By connecting instruction semantics to concrete interactions, the code stream helps align predicted dynamics with the intended task. Since the action expert also conditions on predictive video representations, this alignment provides a pathway for improving language following, complementing the execution results in the main experiments.
Code Expert and Skill Robustness
We further examine whether code-based planning improves the prediction of physical dynamics and skill robustness during precise manipulation. We compare future frames predicted by AW-0, Subgoal, and AW-0 without code for the right wrist camera on Deal Poker, focusing on the final stage in which the right hand inserts a card into a slot. This interaction requires aligning the held card with the narrow opening and guiding it into the slot, making the relative motion of the hand, card, and receptacle critical to an accurate prediction. The three variants receive similar but non-identical initial observations sampled at nearby time steps from the same episode.
The predicted futures reveal distinct differences in the insertion motion: AW-0 depicts the card aligning with and entering the slot more accurately, whereas Subgoal and AW-0 without code produce less coherent insertion sequences, with blur and distortions around the hand and manipulated objects. These observations suggest that explicit skill calls and object arguments help the video expert better predict how the hand and card should move relative to the receptacle. The code expert supports more accurate video generation and therefore provides the action expert with more reliable predictive guidance for precise manipulation.
Hybrid Actions Evaluation
A qualitative example of hybrid evaluation on the “Fold and Stack Towels” task shows a single episode that continues from the upper to the lower row. Dexterous folding motions rely primarily on direct actions from the AW-0 action expert, while code execution handles more structured motions, such as approaching and stacking the towels. These motions provide direct guidance on stacking order and motion direction, improving the semantic reasoning capability of the whole system.
Low-Level Primitives
The parallel-gripper and dexterous-hand embodiments share ten semantic contact verbs but differ in their low-level primitives: end-effector poses for the gripper and relative motion for the dexterous hand. Each API call declares the contact events it produces, allowing a program to be grounded against a demonstration.
Frequently Asked Questions
What is AW-0? AW-0 is an agentic world action model that integrates code-based skill planning with predictive dynamics and continuous control.
How does AW-0 use code during planning? A grounded code expert generates modular skills with explicit parameters and control logic, while code, video, and control are jointly denoised.
What is hybrid execution in AW-0? At each replanning point, a decoded program slice takes over if it passes validity and progress checks; otherwise, the action expert’s action chunk controls the robot.
What datasets are used for pretraining? AW-0 is pretrained on the YAM and Egocentric Human datasets, which together contain 248,642 episodes and 2,913.2 hours of data.