Current video world models miss geometric structure entirely. They are trained on RGB alone, and their training loss provides no signal to resolve the underlying 3D ambiguity of pixel sequences. As a result, models can produce visually plausible rollouts whose geometry is internally incoherent: predicted depth disagrees across views of the same scene, geometric reasoning about occlusion and contact is difficult, and object shape and identity degrade over a rollout. For downstream uses that read off the rollout, this incoherence is the limiting factor.
Contributions
This work introduces a calibration pipeline for recovering metric depth and accurate multi-view extrinsics from any multi-view stereo teleoperation collection with a known URDF. The pipeline combines learned stereo depth with a joint factor graph optimization that pools all episodes of the same physical robot to recover shared kinematic parameters—including joint offsets and hand-eye calibration—alongside per-scene extrinsics. Applied to DROID, this yields DROID-3D, a calibrated 3D corpus providing dense metric depth and recalibrated multi-view extrinsics for over 70,000 episodes.
The work also introduces DepthWorld, a Stable Video Diffusion-based world model that jointly predicts multi-view RGB and depth via spatial latent tiling, leaving the pretrained VAE unchanged.
At equal training budget, depth supervision improves RGB prediction: DepthWorld gains [the reported PSNR improvement in decibels] over an RGB-only baseline with identical architecture and data.
Constructing Large-Scale 3D Datasets for Robot Manipulation
Recent progress in learning-based geometric vision models rests heavily on large-scale datasets providing dense metric depth and calibrated camera poses. The manipulation domain lacks these resources at a comparable scale. Large teleoperation datasets such as DROID provide visual diversity and embodied dynamics, but their raw geometric annotations do not meet the supervision quality required for geometric pretraining.
The proposed approach takes a multi-view stereo teleoperation dataset with a known kinematic model (URDF) as input and produces a calibrated RGB-D corpus in the robot’s coordinate frame. It addresses the domain’s visual and calibration challenges in two stages.
Stage 1: Initial Multi-View Calibration
Stage 1 turns the independently calibrated cameras of each scene into an internally consistent rig placed approximately in the robot’s coordinate frame. This is accurate enough for the robot’s URDF to be rendered into each view and matched against the real image in Stage 2.
Building a 3D-Consistent World Model
DepthWorld aims to generate video rollouts where predicted depth and RGB compose into a single, geometrically consistent 3D world, rather than independent, visually plausible frames. It uses the dense metric depth and calibrated extrinsics of DROID-3D as direct training signals.
Integrating geometric prediction into state-of-the-art video world models presents two primary difficulties. First, modifying a pretrained backbone such as Stable Video Diffusion (SVD)—for example, through channel expansion or dual-branch networks—destroys the strong appearance and motion priors that make the model useful. Second, supervising depth independently per view does not directly encourage 3D consistency across different cameras.
DepthWorld addresses these challenges with two mechanisms. Instead of altering the U-Net architecture, it uses spatial latent tiling to exploit the Variational Autoencoder’s (VAE) ability to encode depth as an image, placing RGB and depth side by side in a wider latent grid. To enhance cross-view consistency, it applies robot-frame auxiliary supervision through a lightweight point-map head that projects predictions into the robot’s physical base frame and supervises them against unified 3D point maps.
Joint RGB-Depth Prediction via Spatial Tiling
Marigold showed that an image-diffusion VAE encodes and decodes depth maps faithfully. Building on this, DepthWorld encodes depth as a separate image alongside RGB and tiles the two in latent space. Each is encoded by the unmodified SVD VAE and placed side by side, so the U-Net sees a wider latent grid covering both modalities. The only network change is extending the spatial position embeddings to cover the wider grid.
Robot-Frame Point-Map Supervision
Per-view depth denoising trains each predicted depth map against its own target independently, with no explicit penalty when predicted depths disagree across different camera views of the same scene. DepthWorld makes cross-view coherence explicit by lifting predictions into the robot base frame and computing the loss there.
Conclusion and Acknowledgments
This work was supported by the European Union’s Horizon Europe projects AGIMUS (No. 101070165), euROBIN (No. 101070596), ERC FRONTIER (No. 101097822), ELIAS (No. 101120237), and ELLIOT (No. 101214398); ČVUT Starting grant “DREAM-ACT” (Project ID CVUT-StG-26-089); and CTU Future Fund (Project ID CVUT-BrF-26-22825M).
The work was also supported by the EU’s Horizon Europe Programme under Grant agreement No. 10113667 (CLARA Project), and was co-funded by the EU from the Operational Programme Jan Amos Komenský (OP JAK), project “Center for Artificial Intelligence and Quantum Computing in System Brain Research,” reg. no. CZ.02.01.01/00/23_029/0008437. Compute resources and infrastructure were supported by the Ministry of Education, Youth and Sports of the Czech Republic through e-INFRA CZ (ID:90254).
Per-View Metric Depth
Each DROID camera—two table-mounted ZED 2 external cameras and a wrist-mounted ZED Mini—is a calibrated stereo pair with a known baseline. Metric depth therefore reduces to predicting per-pixel disparity. For a pixel and its predicted disparity, metric depth is obtained by direct triangulation.
Qualitative Results of DROID-3D Calibration Compared with the Baseline
Additional qualitative comparisons show the poses obtained by the proposed method and the Point World-based baseline.
Depth Preprocessing Rationale
Raw depth maps are converted to VAE-encodable grayscale images through a four-step pipeline designed to keep depth within the distribution on which the Stable Video Diffusion (SVD) VAE was pretrained:
- Depth is downsampled to the working resolution using unprojection and z-buffer min-pooling rather than bilinear interpolation. This preserves foreground geometry and prevents “flying pixel” artifacts at object boundaries.
- Depths are clipped to the 95th percentile of the per-camera distribution: the specified clipping distances for external and wrist cameras. This bounds the VAE input while preserving manipulation-relevant content.
- A log transform compresses the dynamic range, allocating more representational capacity to close-range objects where manipulation precision matters most.
- Values are normalized to the VAE’s expected input range, and the single channel is replicated to three channels.
Calibration Baseline Modifications
To ensure a fair evaluation of the joint factor graph optimization against the per-scene refinement baseline, modeled on PointWorld, the baseline’s depth source was modified. Instead of using FoundationStereo as in the original PointWorld pipeline, the comparison used the proposed global-matching depth. This isolates the performance difference to the structural difference in the optimization—per-scene versus robot-coupled joint graph—rather than a difference in depth predictors.
Geometric Calibration Metrics
Because ground-truth extrinsics are unavailable for DROID, four geometric proxy metrics are used to evaluate calibration quality:
- Ext-Ext (EE) Reprojection Error: Median pixel reprojection error between the two external cameras, computed on a gold-inlier set extracted via MAGSAC enforcing epipolar and Kabsch-based 3D alignment.
- Wrist-Ext (WE) Reprojection Error: Analogous median pixel reprojection error measured between the dynamic wrist camera and the fixed external cameras.
- Robot Depth Error (RD): Median absolute difference in millimeters between the predicted depth map and the depth obtained by rendering the robot URDF using the optimized extrinsics and joint angles.
- Mask IoU: Intersection-over-Union between the URDF-rendered robot silhouette mask and a ground-truth robot segmentation mask produced by RoboEngine on the real image.
Comparison with TesserAct
TesserAct is a text-conditioned, single-view RGB–depth–normal (RGBDN) video model built on CogVideoX-5b-I2V, a 5B-parameter image-to-video diffusion transformer. It is not directly comparable: it has no action conditioning, sees a single camera, and was trained on different data. The work therefore adapts it minimally to the setting and retrains it on the same episodes as DepthWorld at an equal step budget.
Frequently Asked Questions
What is DepthWorld? DepthWorld is a Stable Video Diffusion-based world model that jointly predicts multi-view RGB and depth using spatial latent tiling.
What is DROID-3D? DROID-3D is a calibrated 3D corpus derived from DROID, providing dense metric depth and recalibrated multi-view extrinsics for over 70,000 episodes.
How does DepthWorld encourage consistency across camera views? It uses robot-frame auxiliary supervision, lifting predictions into the robot base frame and comparing them with unified 3D point maps.
How are metric depth maps prepared for the model? They are downsampled with unprojection and z-buffer min-pooling, clipped, log-transformed, normalized to the VAE’s expected input range, and replicated across three channels.