Environments with Single Physics Variation
The Push-T environment consists of a circular pusher (the agent) and a T-shaped block. The world model predicts future scene latents alongside the pusher’s proprioception, conditioned on the pusher’s actions. In planning, each goal is defined by an image and a target agent state. The block’s center of mass (CoM) is randomly sampled. Different CoMs do not change the block’s appearance, but alter how it rotates and slides when pushed, resulting in significantly different states even under identical push actions.
Environments with Multiple Physics Variations
FetchSlide under Multiple Physics Variations
We next investigate whether PhysiCal can calibrate world models in environments where transition dynamics are simultaneously governed by multiple physics properties. To this end, in FetchSlide, we concurrently vary three physics parameters: puck mass, table surface friction, and aerodynamic drag. In this environment, a robot arm strikes a flat puck across a tabletop, where these three properties jointly determine the puck’s post-strike deceleration and sliding distance. The action space is end-effector delta position, which subsequently defines the striking velocity and angle. In planning, we optimize the strike actions so that the puck can stop at the target position specified by a goal image. The context sequence is collected via random actions.
Quantitative results are presented in Table 2. Despite the added complexity of coupled physical factors, +PhysiCal effectively calibrates predictions without requiring analytical physics equations.
Discussion
Conclusion
We believe that world models should go beyond generic visual fidelity toward instance-specific physics-grounded prediction. In this work, we take a step toward this goal by introducing an in-context calibration framework that grounds action-conditioned world models in the actual physics dynamics of novel test scenes. We achieve this by prompting the model with an interaction sequence collected within the test scene, enabling zero-shot calibration without finetuning or analytical physics priors. Evaluations across diverse environments demonstrate that our approach substantially improves predictive accuracy and downstream visual planning.
This work is supported in part by the National Science Foundation under CAREER Award No. 2544116.
Implementation of PhysiCalWorld
Our experiments are implemented in Python 3.10 on Red Hat Enterprise Linux 8.10. We train and evaluate our models using NVIDIA L40S GPUs and NVIDIA RTX PRO 6000 Blackwell GPUs interchangeably.
FetchSlide Physics
At a high level, the interplay between these physics properties can be described as follows. For the same strike action, lower puck mass results in a higher post-strike velocity. The subsequent puck’s deceleration depends jointly on surface friction and aerodynamic drag, where stronger friction and higher fluid density will increase deceleration rate. In addition, since we do not vary the shape of the puck, given the same air density, a puck of larger mass will decelerate slower compared to a lighter puck.
Success-Threshold Selection and Sensitivity
To better understand the planning performance in Reacher and FetchSlide, we additionally present success rate versus error-threshold curves.
In both the Reacher and FetchSlide environments, our method improves performance over the baseline world models (DINO-WM and LeWM), as evidenced by comparing DINO-WM and DINO-WM + ours. Our method improves substantially over the unprivileged baselines in FetchSlide. The gains are modest in the Reacher environment, likely due to its highly constrained state space, which consists of only two rotational joints.
Uncropped BeerPong Visualization
Cropped predicted frames are used for a clearer view. This section provides the uncropped version of the model’s predictions.
Additional Planning Results
We present additional planning comparisons of both DINO-WM- and LeWM-based models. For each example, we use the world model with CEM to optimize a sequence of actions that reaches the goal observation. Planned actions are executed in the simulated environment. We present both the world models’ predicted observations (top row) and the actual execution in the simulated environments (bottom row). Each planning example is followed by the corresponding interaction clip.
Frequently Asked Questions
What does PhysiCal do? It calibrates action-conditioned world models to the physics dynamics of novel test scenes using an interaction sequence collected in that scene.
Does PhysiCal require finetuning or analytical physics priors? No. The framework enables zero-shot calibration without finetuning or analytical physics priors.
Which physical properties vary in FetchSlide? The experiment varies puck mass, table surface friction, and aerodynamic drag.
How does puck mass affect FetchSlide behavior? For the same strike action, a lower-mass puck has a higher post-strike velocity; at the same air density, a larger-mass puck decelerates more slowly.