Trained on heterogeneous human and robot videos, DreamX achieves state-of-the-art performance among latent action models on LARYBench and enables any-to-any video transfer and human-to-robot action transfer without paired training data. As a downstream application, learned latent actions can serve as prediction targets for an observation-conditioned robot policy, without requiring future-video generation during execution. Together, these results suggest that DreamX learns a shared, transferable latent action space that supports motion transfer across embodiments and can also serve as a basis for downstream policy learning.
Introduction
In summary, the contributions are as follows:
- DreamX is a framework that uses the spatiotemporal and generative priors of a pretrained video diffusion model to learn universal latent actions. The resulting representations achieve state-of-the-art performance among latent action models on LARYBench.
- The learned latent space supports any-to-any video transfer and human-to-robot action transfer without paired source–target training data. Compact latent-action codes can also serve as prediction targets for downstream policy learning.
Preliminaries: Video Diffusion Models
The model predicts the flow velocity.
By optimizing this objective over large-scale video data, the model acquires a generative prior over scene appearance and plausible dynamics. When the initial frame supplies the scene’s appearance, additional conditioning steers how the scene evolves. DreamX uses this property to steer future-frame reconstruction with latent actions; the reconstruction objective supervises what the latent actions encode.
Learning Universal Latent Actions from Video Diffusion Priors
DreamX adapts a pretrained video diffusion backbone for two complementary roles: encoding observed dynamics into latent actions and supervising them through future-frame reconstruction to encourage separation of motion from visual appearance. Learnable queries read out the video representations to summarize observed dynamics as latent actions, while the decoder reconstructs the future from these latent actions and the initial frame. Using a video diffusion backbone for both the encoder and decoder allows its generative prior to account for visual appearance, encouraging the latent action tokens to focus on motion.
Video Transfer
DreamX is evaluated on transferring source motion across scenes and embodiments while preserving the appearance specified by a target initial frame. The evaluation examines any-to-any video transfer and quantitatively evaluates zero-shot human-to-robot transfer.
Motion transfer is shown across environments for the same robot, between different robot embodiments, and from humans to robots, with a DAVIS example extending beyond manipulation. DreamX reproduces the source motion while preserving the target scene and embodiment. These results support the universality of the learned latent actions across embodiments and environments, extending to general-domain videos.
Conclusion
DreamX leverages pretrained video diffusion priors to learn universal latent actions. Its spatiotemporal representations provide action-relevant dynamics, while its generative prior supervises latent-action learning through future-frame reconstruction. DreamX achieves state-of-the-art performance among latent action models on LARYBench and enables any-to-any video transfer and human-to-robot action transfer without paired training data. Predicting these compact latent actions also supports effective and efficient policy learning from human and robot demonstrations. Together, these results highlight pretrained video models as a foundation for learning reusable action representations across embodiments.
AI Use Statement
Generative AI tools assisted with manuscript drafting and editing for clarity and readability, code editing during implementation, and discussions during the research process. The authors formulated, evaluated, and ultimately determined the research ideas, methodological choices, experimental designs, and interpretation of results. The authors reviewed all AI-assisted text and code, and code changes were tested for correctness. The authors take responsibility for the final content of this work, including text, claims, code, and artifacts produced with the aid of generative AI.
Ethics Statement
All datasets used in this work are publicly available, including the human egocentric and robot demonstration datasets listed in the appendix. Human videos may contain identifiable people or surroundings, so any redistributed examples should follow the source datasets’ usage terms and privacy requirements. Video-transfer outputs are synthetic and should be identified as such. Cross-embodiment transfer and robot policies may fail outside the evaluated settings and require safety validation before deployment.
Real-World Robot Setup
Both action-transfer and policy experiments use a bimanual YAM robot. The action space consists of 14-dimensional absolute commands, comprising six joint positions and one gripper command per arm. An Intel RealSense D405 camera mounted in a top-view position provides observations.
Action Transfer Details
Latent actions are extracted from human demonstrations using the frozen DreamX encoder and decoded into robot commands. The resulting trajectory is executed open-loop, without visual feedback, conditioned on the human latent actions and the robot’s initial proprioceptive state.
Policy Evaluation Details
Three drawer-closing configurations are evaluated: only the upper drawer open, only the lower drawer open, and both drawers open. All configurations use the instruction “close the drawer.” When both drawers are open, the robot must close the upper drawer before the lower drawer. A fixed top-view RGB camera observes the workspace.
LARYBench Evaluation Details
All representation models are frozen, and a common attentive probe is trained for action regression and classification. Each input token is projected to 768 dimensions through a linear layer, layer normalization, and GELU. For representations spanning multiple transitions, learned temporal embeddings identify the transition positions before the temporal and token dimensions are flattened. A learned-query attentive pooler aggregates the tokens into a single feature for the regression or classification head.
Zero-Shot Human-to-Robot Transfer Details
The evaluation uses 1,008 clips spanning 21 tasks from HumanAndRobot, selecting 48 clips per task. Each method receives a human source video and the initial frame of its paired robot recording. The remaining robot frames serve only as evaluation references; no paired human–robot data is used for training. DreamX is compared with AdaWorld and VideoWorld 2 using their native generation horizons.
For a common evaluation horizon, the first 20 frames of each output are retained and the conditioning frame is excluded, yielding 19 generated frames per clip. The reported metrics are FVD using a Kinetics-400-pretrained I3D network, VGG-based LPIPS, PSNR, and SSIM against the corresponding robot reference videos.
Policy Evaluation
Reaching and pushing completion are reported separately. Averaging each pair gives the per-configuration success rates: 50.0% for Mimic-Video in all configurations, and 90.0%, 100.0%, and 100.0% for DreamX.
Representation Analysis
The analysis examines whether latent actions exhibit common directional structure across robot embodiments using 2,362 RoboCOIN clips spanning ten robot types and six end-effector translation directions.
A t-SNE visualization of 60 direction-conditioned latent-action centroids shows that centroids for the same direction exhibit similar organization across robot types. This aggregate analysis supports shared directional structure across embodiments.
Any-to-Any Video Transfer
Additional any-to-any video transfer examples cover different embodiments and environments, complementing the examples in the main paper.
Human-to-Robot Action Transfer
Additional human-to-robot action transfer examples cover writing and single-arm manipulation. The experimental setup is described in the action transfer details.
Policy Rollouts
Illustrative real-world drawer-closing rollouts cover the upper-only, lower-only, and both-open configurations.
Query-to-Video Attention
Additional query-to-video attention visualizations cover human and robot demonstration sequences. Query-to-video attention is averaged over all heads, the four query slots within each temporal group, and the final seven transformer blocks.
Frequently Asked Questions
What does DreamX learn from video? DreamX learns universal latent actions that represent motion and can be transferred across scenes and embodiments.
Does human-to-robot transfer require paired training data? No. DreamX enables human-to-robot action transfer without paired source–target training data.
How are DreamX latent actions used for robot policies? They serve as prediction targets for an observation-conditioned robot policy, without requiring future-video generation during execution.
What robot setup is used for real-world experiments? The experiments use a bimanual YAM robot with 14-dimensional absolute action commands and a top-view Intel RealSense D405 camera.