Introduction
The defining breakthroughs in large language models were precipitated by web-scale training on diverse text datasets, shattering performance barriers by learning a unified representation of human language. Today, as action-conditioned video world models continue to struggle with physical inconsistency, the robotics and vision communities face a critical question: can generative video models learn generalizable physical priors through cross-embodiment data scaling, or does the heterogeneous nature of different robot morphologies demand agent-specific architectures? This work investigates how video world models can bridge the gap between diverse embodiments and analyzes the structural bottlenecks that must be overcome.
Beyond Zero-Shot: Data-Efficient Adaptation to Target Morphologies
Through broad-scale training on diverse robot video data, CLAP’s models learn generalizable priors that deliver high-accuracy dynamics prediction. Nonetheless, their zero-shot prediction accuracy can be improved by finetuning them on target embodiments, spanning pretrained embodiments that have been seen during training and novel embodiments. Via finetuning, these models can refine their learned priors to better align them with target embodiments.
Novel Embodiments
Beyond pretrained embodiments, CLAP facilitates data-efficient adaptation of cross-embodiment models to novel robot morphologies that are markedly different from those seen during training, such as bimanual robots and humanoids, which typically feature higher-dimensional action spaces.
Due to this action-space disparity, model finetuning requires substituting the model’s action head with a new one compatible with the target embodiment. Crucially, CLAP retains all other components of the cross-embodiment video model to preserve rich spatiotemporal priors. During finetuning, CLAP aligns these underlying priors with the target morphology, enabling high-fidelity action-conditioned future prediction.
Beyond Zero-Shot: Does CLAP Facilitate Data-Efficient Adaptation to Target Embodiments?
We evaluate the effectiveness of CLAP as a foundation for few-shot adaptation of video world models to target embodiments. We ground our evaluations along two primary axes: pretrained embodiments and novel embodiments. In each setting, we drop incompatible components of the action head to match the native end-effector action space of the target embodiment but retain all other parameters.
Preliminaries
Video world models predict future outcomes conditioned on per-frame actions, starting from an initial camera observation. Furthermore, to capture temporal context such as robot velocities that dictate future evolution, video world models are often conditioned on a history of past observations.
The omitted formula defines the history and prediction horizons used by the model.
Additional Results on Latent Action Models
We train latent action models on our cross-embodiment dataset and individual single-embodiment subdatasets, including Bridge and DROID, and evaluate the consistency of the proxy actions relative to the ground-truth trajectory.
Using latent action models as inverse-dynamics models, we extract latent actions for unseen trajectories and reconstruct the ground-truth trajectories using the computed latent actions. We benchmark these models against the DreamDojo inverse-dynamics-model baseline on the validation and test splits of the in-domain datasets, including Bridge, DROID, and OXE-Mix.
We also evaluate the models on the following held-out datasets, representing unseen data:
austin_sailorutaustin_mutexberkeley_ur5stanford_hydra
These datasets include robot morphologies, such as the UR5 arm, that were not seen during training.
Nuanced Summary
This section discusses the nuances surrounding CLAP’s novelty, broader potential impact, and scope of claims.
Conclusion, Limitations, and Future Work
Ultimately, this work establishes cross-embodiment learning as a catalyst for breakthroughs in video world modeling, mirroring the paradigm shifts observed in large language models.
Safety, Data Privacy, and Consent
This paper presents work advancing the foundations of artificial intelligence in robotics. Because this work is primarily computational and theoretical, it does not involve human participants, live user studies, or personally identifiable information; consequently, Institutional Review Board approval was not required.
All experiments were conducted using publicly available benchmarks and open-source datasets in accordance with their respective licenses. While broader applications of artificial intelligence in robotics carry potential safety and societal implications, the authors do not foresee any direct, malicious, or high-risk negative societal consequences uniquely tied to this fundamental algorithmic research.
Acknowledgments
The authors would like to thank Prof. Anirudha Majumdar, Mingtong Zhang, and members of the Princeton IRoM lab for their patience, insightful discussions, and support.
Frequently Asked Questions
What is CLAP? CLAP is a cross-embodiment video world model framework for learning generalizable physical priors from diverse robot video data.
What do video world models predict? They predict future outcomes conditioned on per-frame actions and an initial camera observation, often using a history of past observations to capture temporal context.
How does CLAP adapt to novel robot embodiments? CLAP substitutes the action head with one compatible with the target embodiment while retaining the other components of the pretrained video model.
Which datasets are used to evaluate latent action models? The evaluation includes Bridge, DROID, OXE-Mix, and held-out datasets including austin_sailor, utaustin_mutex, berkeley_ur5, and stanford_hydra.