T2-z augments robotic world models with environment-latent conditioning. It alternates between world-model optimization, which learns action-conditioned dynamics, and environment-latent optimization, which captures hidden environment factors. For a new environment, the world model is frozen and only the latent is optimized. Across seven simulation and four real-world tasks, T2-z outperforms baselines in video prediction and action selection.
Method
T2-z alternates between optimizing the world model and optimizing an environment latent. The world model learns action-conditioned dynamics, while the latent captures hidden environmental factors. When adapting to a new environment, the world model remains frozen and only the latent is optimized.
Simulator and Real-World Setup
The custom tasks use a LIBERO-based simulator with a Franka Panda arm. Simulated RGB videos are recorded at 20 frames per second and [image resolution not specified in the provided text]. The action representation has 14 channels.
Standard World Model Baseline
The Standard World Model baseline fine-tunes the BWM-initialized DiT and action encoder on pooled task trajectories. It does not use an environment latent or test-time adaptation.
The baseline uses AdamW with a model learning rate of [value not specified in the provided text], weight decay of [value not specified in the provided text], 100 warmup updates, gradient clipping of [value not specified in the provided text], and bfloat16.
Computational Efficiency
Computational efficiency is profiled in a representative Friction setting on one H200 GPU. Each method is warmed up for two optimizer steps and timed for ten further steps.
The reported measure is the median compute-only optimizer time per update, excluding dataset reads, along with peak allocated GPU memory; it is not full-run training time. Each measured update has six loss-bearing video chunks, and method-specific conditioning computation is included.
The world-model and latent updates in T2-z are separate phases, so they are reported separately. LoRA TTA starts from the Standard World Model and has no distinct world-model training row.
Conclusion
T2-z augments robotic world models with environment-latent conditioning. It alternates between world-model optimization that learns action-conditioned dynamics and environment-latent optimization that captures hidden environment factors. For a new environment, the world model is frozen and only the latent is optimized. Across seven simulation and four real-world tasks, T2-z outperforms baselines in video prediction and action selection.
Frequently Asked Questions
What does T2-z optimize when adapting to a new environment? It optimizes the environment latent while keeping the world model frozen.
What does the world-model optimization learn? It learns action-conditioned dynamics.
What does the environment latent capture? It captures hidden environment factors.
How many tasks are reported across simulation and the real world? The paper reports seven simulation tasks and four real-world tasks.