Scaling World Action Models fundamentally depends on scaling embodied experience rather than robot trajectories alone. However, embodied experience is inherently heterogeneous across egocentric human videos, handheld-gripper demonstrations, and robot trajectories. These sources differ substantially in observation modalities, action representations, temporal granularity, and supervision fidelity, making unified large-scale pretraining fundamentally challenging.
To address this challenge, we build a unified embodied data infrastructure that automatically transforms heterogeneous embodied experience into a common action-level video–state–action representation. The proposed infrastructure consists of three tightly coupled components: multi-source embodied experience collection, a unified embodied data engine, and progressive supervision with multi-source data balancing. Together, they produce a scalable pretraining corpus containing more than thousands of hours of embodied experience covering thousands of interaction skills, providing the data foundation for World Action Modeling.
Data Infrastructure
Scaling World Action Models fundamentally depends on scaling embodied experience rather than robot trajectories alone. However, embodied experience is inherently heterogeneous across egocentric human videos, handheld-gripper demonstrations, and robot trajectories. These sources differ substantially in observation modalities, action representations, temporal granularity, and supervision fidelity, making unified large-scale pretraining fundamentally challenging.
To address this challenge, we build a unified embodied data infrastructure that automatically transforms heterogeneous embodied experience into a common action-level video–state–action representation. The proposed infrastructure consists of three tightly coupled components:
- Multi-source embodied experience collection
- A unified embodied data engine
- Progressive supervision with multi-source data balancing
Together, they produce a scalable pretraining corpus containing more than thousands of hours of embodied experience covering thousands of interaction skills, providing the data foundation for World Action Modeling.
Multi-Source Embodied Experience
The three data sources provide complementary supervision. Egocentric human videos offer the broadest diversity of real-world interactions, object-centric manipulation skills, and long-horizon task compositions. Handheld-gripper and wearable demonstrations provide structured end-effector trajectories that naturally bridge human interactions and robot-compatible control. Robot trajectories provide precise embodiment-specific states and directly executable actions for downstream policy learning.
Rather than being redundant, these sources complement each other:
- Human videos provide scalable interaction knowledge.
- Handheld-gripper demonstrations reduce the embodiment gap between humans and robots.
- Robot trajectories provide executable control supervision.
Together, they establish a natural supervision hierarchy for Progressive Embodied Pretraining.
Unified Embodied Data Engine
Although the three data sources provide complementary supervision, their native formats are fundamentally different. Egocentric human videos provide visual observations with weak language supervision, handheld-gripper demonstrations provide structured end-effector trajectories and gripper states, while robot datasets differ substantially in embodiment-specific states, action spaces, coordinate systems, and control frequencies. Directly combining these heterogeneous sources would result in inconsistent supervision and unstable optimization.
To unify heterogeneous embodied experience, we build a unified embodied data engine that converts all data into a common action-level trajectory representation consisting of:
- Language instructions
- Embodiment identities
- Visual observations
- States
- Actions
- Semantic metadata
For each embodiment, state and action streams are temporally aligned, normalized, and organized under embodiment-specific canonical definitions, while validity masks handle variable-dimensional actions without forcing heterogeneous embodiments into a shared physical action space. This unified representation serves as a standardized interface between heterogeneous embodied data and the World Action Modeling objective.
Unlike egocentric human videos, handheld-gripper demonstrations and robot trajectories already provide structured control supervision. Their processing therefore mainly focuses on temporal alignment, action normalization, and trajectory quality filtering. Gripper-state transitions are used to refine action boundaries, while camera-shake filtering, stationary end-effector rejection, and abnormal control filtering further improve data quality.
After processing, all data sources are converted into temporally aligned action-level video–state–action trajectories under a unified semantic taxonomy, enabling scalable World Action Modeling across heterogeneous embodiments.
Progressive Supervision and Data Balancing
The processed corpus forms a continuum of supervision with progressively increasing action fidelity rather than a simple division between labeled and unlabeled data. Large-scale egocentric human videos first provide broad interaction dynamics and weak action supervision through pseudo actions generated by a frozen Latent Action Model. Human videos with reconstructed 3D hand trajectories, handheld-gripper demonstrations, and heterogeneous robot trajectories subsequently provide continuous real-action supervision for visual-action alignment. Finally, high-quality robot trajectories specialize the model for embodiment-specific action prediction and closed-loop policy execution.
This supervision hierarchy naturally matches the three-stage training curriculum of Riemann-1.0, enabling the model to progressively transfer from open-world interaction knowledge to executable robot control.
The raw corpus exhibits severe long-tail distributions across data sources, scenes, tasks, skills, objects, and robot embodiments. Rather than balancing raw data volume, we construct a unified semantic taxonomy and perform semantic-aware sampling across multiple semantic dimensions. This strategy preserves the scale advantage of large human datasets while ensuring sufficient exposure to long-tail manipulation skills and low-resource robot embodiments.
Overall, our data infrastructure transforms heterogeneous embodied experience into a scalable supervision source for World Action Modeling, enabling World Action Models to continuously improve by scaling embodied experience rather than robot trajectories alone.
Training Recipe
Rima-WAM is trained with a three-stage curriculum that progressively shifts supervision from video-only pseudo actions to real action trajectories. This design is motivated by a practical data imbalance: video-only data is much larger and visually diverse, but it lacks executable robot actions, whereas real action datasets are smaller but provide direct supervision over robot behavior.
The three stages are:
- The first stage uses a frozen Latent Action Model to train large-scale unlabeled videos with pseudo-action supervision.
- The second stage grounds the model on a mixed corpus of UMI demonstrations, robot trajectories, and human videos with 3D hand annotations.
- The third stage continues on high-quality robot-only data to sharpen executable control.
Across all stages, the model optimizes the same action-video objective.
Pretraining
The first stage uses only unlabeled human manipulation videos. Since these videos provide rich visual interaction but lack executable action labels, we first train a Latent Action Model and then freeze it as a pseudo-action annotator.
This stage aims to expose WAM to a large amount of manipulation dynamics before real robot action labels become available. The Latent Action Model provides this bridge by forcing the motion between two frames to pass through a compact latent bottleneck, so that the inferred latent code captures the underlying visual transition that an action would have induced and can serve as pseudo-action supervision for WAM training.
The KL term is deliberately weak: its role is to regularize the latent distribution without collapsing the transition code into an overly generic prior. After training, we use the posterior mean the average latent representation predicted by the model rather than a sampled latent as the deterministic pseudo action, eliminating sampling noise when annotating large-scale video corpora.
Real-World Experiments
After large-scale pretraining, we further adapt Riemann-1.0 to real-world deployment through real-robot post-training. We build four representative manipulation scenarios on the Tianji Marvin dual-arm robot:
- Ordered cube stacking
- Clothes folding
- Desk organization
- Kitchen organization
These tasks cover order-constrained rigid-object stacking, deformable-object manipulation, long-horizon tabletop rearrangement, and long-horizon kitchen storage, forming a compact but challenging suite for household robot deployment. We collect 15 demonstrations for each task using human teleoperation.
Ordered Cube Stacking
In ordered cube stacking, the robot follows a language instruction to identify colored cubes and assemble a four-layer stack following a specified color order. Four-layer stacking sharply increases the difficulty: small alignment errors at lower layers are amplified by later placements, and a single misaligned layer can destabilize the entire stack.
Therefore, this task requires accurate visual recognition of object color and pose, planning of the grasping sequence, and precise pick-and-place execution.
Clothes Folding
In clothes folding, the robot manipulates a deformable garment rather than a rigid object. Clothes can wrinkle, stretch, and drift in pose during contact, so the robot must continuously adapt its action strategy based on the observed state.
This task stresses deformable-object perception, bimanual coordination, and fine-grained contact control.
Desk Organization
In desk organization, the robot must identify and rearrange multiple scattered desktop objects into their designated locations. A key challenge is inserting slender objects such as pens into a pen holder.
This requires accurate perception of the pen pose and the holder opening, exploiting the rim of the holder to reorient the pen toward a vertical pose, and then controlling the gripper to complete a precise insertion.
Kitchen Organization
In kitchen organization, the robot must complete an end-to-end tidying task in a kitchen scene. It first localizes and grasps utensils such as forks and spoons, then places them back into the target storage area.
It must also grasp yellow and red plates and insert them vertically into a drying rack, stack bowls, and place the bowl stack on the upper layer of the rack, requiring long-horizon sequencing and stable placement under cluttered conditions.
We compare Riemann-1.0 with representative open-source VLA and WAM baselines, including DreamZero, a World Model baseline, LingBot-VLA, an additional comparison model, LingBot-VA, and G0.5, under the same real-world manipulation setting.
For DreamZero, we reproduce the model using the official open-source implementation, training a 5B-scale model on our collected large-scale manipulation dataset, denoted as DreamZero*.
Action-Conditioned Visual Rollout as a Multi-Embodiment Simulator
Riemann-1.0 can also be used as an action-conditioned visual simulator. In this mode, the model is not asked to only output an executable action. Instead, it receives the current visual observation, task prompt, robot state, and a candidate future action trajectory, and predicts the visual consequences of executing the trajectory.
The action trajectory is represented with the same embodiment-specific action interface used during policy training, then embedded as action tokens and inserted into the causal action-video sequence. These action tokens condition the visual-latent denoising process, so the generated frames are tied to the robot motion rather than sampled as an unconditional future video.
At inference time, we first encode the current observation into the VAE latent space and use it as the clean visual context. The future action chunk is either provided by the policy head, sampled from a candidate plan, or taken from a recorded trajectory for analysis.
Conditioned on the prompt, state, clean visual context, and action tokens, the latent head rolls out future visual latents; these latents are then decoded back into RGB video. For longer horizons, the generated or observed visual output can be appended back to the context and the same procedure can be repeated autoregressively.
This gives Riemann-1.0 a data-driven simulator interface: actions are treated as controllable inputs, while future camera observations are produced as the simulated consequences.
Frequently Asked Questions
What is the purpose of Riemann-1.0’s embodied data infrastructure? It transforms heterogeneous embodied experience into a common action-level video–state–action representation for World Action Modeling.
Which data sources are used for embodied pretraining? The infrastructure uses egocentric human videos, handheld-gripper and wearable demonstrations, and robot trajectories.
What is the training curriculum for Rima-WAM? The three stages progress from video-only pseudo-action supervision to mixed real-action data and finally high-quality robot-only data.
How is Riemann-1.0 used as a visual simulator? It receives visual observations, task prompts, robot states, and candidate action trajectories, then predicts the visual consequences of executing those actions.