Method
The training and planning pipeline of AD-WM uses residual latent prediction and action-recovery objectives to shape action-discriminative transitions. At test time, model predictive control compares candidate action sequences using the learned dynamics, while the auxiliary heads are discarded. The task is formulated first, followed by the model, training objectives, planning procedure, and diagnostics for evaluating candidate selection.
Residual Latent Prediction
Successive visual representations often share substantial state information. The current representation carries this shared information, while the predictor estimates the remaining change. Given a learned action embedding, AD-WM predicts:
The model predicts the residual change in the latent representation conditioned on the current representation and action embedding.
The prediction is supervised by the encoded successor:
The predicted residual is trained against the difference between the encoded successor and the current representation.
Experiments
The experiments address four questions:
- Q1: How well does AD-WM perform across simulation tasks and configuration shifts?
- Q2: How do residual prediction and action recovery contribute to control performance?
- Q3: Which prediction and selection metrics best reflect closed-loop success?
- Q4: Can AD-WM transfer to a real robot without lab-specific adaptation?
Conclusion and Limitations
AD-WM is an action-discriminative world model for counterfactual model predictive control. It combines residual prediction with predictor-level action recovery to preserve action information without changing the planner.
Experiments show gains over matched LeWM in four of five simulation environments and transfer to Franka without lab-specific adaptation. Cube ablations identify residual prediction and mutual information as the largest contributors, while inverse dynamics has a smaller, weight-dependent effect on mean success. Cube diagnostics show that elite regret tracks success more closely than factual error or whole-bank ranking.
Diagnostics remain limited to Cube, and Scene’s gain is unresolved over three seeds. External comparisons differ in inference and checkpoint counts. Simulation and robot post-training use different auxiliary weights. Robot evaluation uses manual subgoals and non-randomized trials with one site, camera, and backbone. Future work examines broader settings and reduces reliance on manual subgoals.
Frequently Asked Questions
What is AD-WM? AD-WM is an action-discriminative world model for counterfactual model predictive control.
Which objectives does AD-WM combine? It combines residual latent prediction with predictor-level action recovery.
Where did AD-WM show gains? It improved performance over matched LeWM in four of five simulation environments and transferred to Franka without lab-specific adaptation.
What are the main limitations? Diagnostics remain limited to Cube, while the robot evaluation used manual subgoals, non-randomized trials, one site, one camera, and one backbone.