One Video World Model Uses Robot Renderings to Predict Across Embodiments

One Video World Model Uses Robot Renderings to Predict Across Embodiments

Yash Jangir, Karan Mirakhor, Renos Zabounidis, Muhammad Zubair Irshad, Sergey Zakharov +2 more

7 min read•Nov 19, 2026

The key idea is to provide the robot’s intended motion explicitly so the video model can focus on predicting its effects on the environment. First, the camera pose relative to the robot base is recovered. The robot is then rendered along the supplied configuration trajectory from this viewpoint. Doppel uses a render-enriched action conditioning adapter to combine these renderings with numerical joint and gripper trajectories. The renderings show where the robot moves in the image, while the numerical inputs retain configuration information that may be difficult to see. This representation conditions Doppel, a single image-to-video model shared across embodiments.

Doppel

Doppel requires renderings of the robot from the same viewpoint as the recorded observations. Source datasets, however, have missing or inconsistent camera extrinsics. The camera-to-robot transform is recovered by optimizing its six degrees of freedom to align rendered robot trajectories with the recorded video using RGB, optical flow, and DINOv2 features. The robot renderer is kept fixed; only the camera pose is optimized using an analysis-by-synthesis approach.

Rendering-enriched dense action representation

Robot renderings and numerical actions are combined to produce one conditioning vector for each video token. Renderings describe the robot’s motion in image space, while numerical actions provide joint and gripper information that is hidden by occlusion or ambiguous from a single view.

Architecture of the rendering-enriched action conditioning

Discussion, limitations, and conclusion

The method requires accurate robot geometry, a URDF, and camera calibration to construct the renderings. Although missing extrinsics can be recovered in offline datasets, unknown or rapidly changing viewpoints remain a limitation. Camera-aware or camera-invariant representations could reduce this dependence.

The method also conditions on configuration trajectories. Handling imperfect geometry or lower-level control commands would require modeling uncertainty in robot motion and execution. The evaluation covers embodiments seen during training and one camera per trajectory. Transfer to unseen embodiments, multi-view prediction, and longer-horizon prediction are left to future work. Combining pretrained video models with learned 3D representations may further reduce the need for explicit pixel-level alignment.

Website and video

Additional qualitative results are available on the project webpage, including extended video comparisons across all eight datasets, long-horizon rollouts, examples of trajectory and environment generalization, and side-by-side comparisons with baselines. The authors encourage readers to view the webpage for the full video results, which are difficult to represent comprehensively in static paper figures.

Rendering pipeline

The robot is rendered along the supplied joint and gripper trajectory using an articulated Gaussian robot model and the target camera calibration. The renderer is fixed during world-model training. RGB observations, robot configurations, and rendered frames are constructed using the same sampled source-frame indices, ensuring temporal alignment across all conditioning signals. When constructing training windows, cached renderings are also checked to ensure they correspond to the same temporal sampling grid.

The robot-only renderings encode the embodiment’s geometry and motion directly in image coordinates; they contain no future scene or object information. Representative alignments between rendered robot trajectories and corresponding RGB observations are shown across the three datasets.

For each embodiment, the method uses a differentiable robot-rendering framework. Starting from the robot’s URDF, an articulated Gaussian model is trained using the original renderer architecture and training procedure. A separate renderer is trained for each robot embodiment, and its parameters are kept frozen when training Doppel.

The bimanual YAM arms in ABC are rendered with a DrRobot model trained for the YAM in the same way. The Vega-1 humanoid and its Sharpa hands in T-Rex are rendered from the robot’s URDF meshes using the dataset’s released head-camera calibration. People are rendered from fitted body and hand meshes: MANO hands for GigaHands, SMPL-X for OakInk2, and SMPL-H for Ego-Exo4D, using the meshes of Maddukuri and Pavlakos. Each mesh is rasterized through the dataset’s camera model, including lens distortion. All renderings use a black background and contain only the embodiment.

Rendered robot trajectories aligned with recorded observations

Adapter architecture details

The render–action adapter contains about [the parameter count is not specified in the provided text] parameters and is shared across all embodiments.

Robot-only renderings are sparse, with most spatial locations containing empty background. The rendering sequence is first encoded using the same frozen causal VAE of Wan 2.1. Before applying the trainable render encoder, the per-sample, per-channel spatiotemporal mean is subtracted.

This prevents empty backgrounds from making the render tokens very similar. If this mean subtraction is skipped, the adapter can collapse to zero weights. The operation is computed in [the representation or space is not specified in the provided text], since the robot signal can be small relative to the shared latent response of the empty rendering background.

Baseline re-implementation

The action-only baseline follows the conditioning mechanism of EgoWM. Starting from the same Wan 2.1 image-to-video backbone used by Doppel, it removes the rendering encoder and render–action fusion modules entirely. The numerical robot configuration at each timestep is normalized using the same per-embodiment statistics as Doppel, projected with an MLP followed by a 1D temporal convolution, and injected through the backbone’s timestep-modulation pathway. Thus, the baseline receives the same numerical robot trajectory as Doppel but no camera-aligned robot renderings.

All remaining components are kept identical to the corresponding Doppel configuration, including backbone initialization, training clips, clip construction, optimization schedule, conditioning dropout, and training budget. All models are evaluated on exactly the same validation and test splits. The complete single-embodiment training recipe is referenced in the paper, but its table data are not included in the provided text. This baseline therefore provides a controlled comparison between numerical action-only conditioning and Doppel conditioning.

Evaluation metrics

Generated videos are evaluated against reference videos at [the resolution is not specified in the provided text]. All metrics exclude the conditioning frame and are computed over the sixteen predicted frames. Unless otherwise specified, scores are first averaged over time within each clip and then averaged across clips with equal weight. Undefined scores are omitted.

The reported metrics are peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and learned perceptual image patch similarity (LPIPS). PSNR measures pixel-level reconstruction fidelity, SSIM measures local agreement in luminance, contrast, and structure, and LPIPS measures perceptual discrepancy using pretrained AlexNet features. Each metric is computed independently for every predicted frame and averaged over the prediction horizon. Higher PSNR and SSIM are better, while lower LPIPS is better.

Results

Additional qualitative results are provided across all experiments. The paper also includes an example of trajectory generalization comparing Doppel with the baseline.

Frequently Asked Questions

How does Doppel condition video generation on robot motion? It combines robot renderings with numerical joint and gripper trajectories to create conditioning vectors for video tokens.

Why does Doppel use both renderings and numerical actions? Renderings show robot motion in image space, while numerical actions preserve joint and gripper information that may be occluded or ambiguous in a single view.

What does the rendering pipeline require? It requires robot geometry, a URDF, and camera calibration; a separate articulated Gaussian renderer is trained for each embodiment and kept frozen during Doppel training.

What limitations remain? The method depends on accurate geometry and calibration, and its evaluation covers training-seen embodiments with one camera per trajectory. Transfer to unseen embodiments, multi-view prediction, and longer-horizon prediction are left for future work.