Contact-rich manipulation poses a core and persistent challenge in robotic manipulation, essential for practical deployment. Unlike free-space manipulation, these tasks rely on local interaction states, including deformation, pressure, slip, and friction. These states are often weakly visible, transient, or occluded in visual observations, making vision-centric policies unreliable when execution demands tactile-informed adjustments.
To address this issue, the key insight is to couple action prediction with tactile evolution, enabling the policy to leverage tactile changes during contact phases. Recently, World Action Models (WAMs) have provided the ability to predict world dynamics by coupling action prediction and video prediction. Building on world action models, we propose VT-WAM, a visual-tactile world action model that jointly learns future visual prediction, tactile deformation prediction, and action prediction within a unified flow matching framework.
In particular, VT-WAM has two core modules that make tactile dynamics useful for action prediction. Asymmetric MoT Attention routes action tokens to a first-frame visual anchor for scene context and to the full tactile sequence for contact evolution. This enables visual-cache inference mode without discarding tactile dynamics needed for contact phases. Contact-gated AVTAG further reduces visual-dominance bias by applying a training-only hinge ranking loss that encourages action queries to attend to tactile evidence during contact phases. This auxiliary guidance makes the model rely more on tactile dynamics when contact information is physically informative, without changing the inference-time architecture.
We evaluate VT-WAM on six real-world contact-rich tasks, covering surface-interaction and constrained insertion regimes. VT-WAM achieves a 71.67% success rate and outperforms the baseline Fast-WAM by 26.67%. Detailed ablation studies of tactile dynamics modeling methods and attention guidance demonstrate the effectiveness of our core designs in contact-rich tasks.
Main Contributions
- We formulate VT-WAM to couple tactile deformation dynamics with action prediction through joint visual-tactile-action flow matching.
- We introduce Asymmetric MoT Attention and AVTAG to enable a visual anchor, temporal tactile dynamics, and contact-phase tactile guidance.
- We validate VT-WAM on six real-world tasks, reaching 71.67% average success, 26.67% above Fast-WAM; ablations confirm both designs.
Experiments
In this section, we first describe the experimental setup, including the robotic platform, implementation details, baselines, benchmark tasks, and evaluation metrics. We then evaluate VT-WAM on six contact-rich manipulation tasks, analyze visual-tactile prediction quality, and conduct ablation studies to quantify the contribution of key components.
Main Results
Table I reports performance across the six contact-rich tasks. VT-WAM achieves the highest success rate among all evaluated methods. Compared with the strongest baseline Fast-WAM, VT-WAM improves the success rate from 45.00% to 71.67%, corresponding to an absolute gain of 26.67%.
In constrained insertion tasks, success depends on fine alignment rather than sustained surface coverage. Across insert plug, swipe card, and insert tube, VT-WAM achieves 61.67% success rate, compared with 38.33% for OmniVTLA and 33.33% for Fast-WAM. These results suggest that tactile dynamics are also useful when the robot must correct small pose errors under tight geometric constraints. The improvement is especially clear on the insert tube, where the transparent tube makes visual alignment unreliable and successful execution requires contact-informed correction. Together with the surface-interaction results, this shows that coupling tactile deformation dynamics with action prediction improves success rates on contact-rich tasks.
Conclusion
We introduce VT-WAM, a Visual-Tactile World Action Model for contact-rich manipulation. VT-WAM extends the world action model by learning tactile deformation as temporal interaction dynamics together with action prediction, rather than using tactile observations only as auxiliary policy inputs. Real-world experiments across surface-interaction and constrained insertion tasks show that VT-WAM consistently improves over visual-only and tactile-input baselines, while tactile prediction analysis and ablations further support the importance of tactile dynamics modeling and contact-phase tactile use. These results indicate that modeling tactile deformation as interaction dynamics provides an effective way to improve action prediction in contact-rich tasks. While this work focuses on individual-task specialization for precise tactile modeling, multi-task training remains unexplored. Future research on multi-task training and scaling laws is a promising direction.
Frequently Asked Questions
How does VT-WAM differ from standard visual-only world models? VT-WAM jointly learns tactile deformation dynamics alongside visual and action prediction, rather than using tactile data only as auxiliary policy inputs, enabling the model to leverage contact-phase tactile changes for more reliable action prediction.
What are the two core modules that make tactile dynamics useful for action prediction? Asymmetric MoT Attention routes action tokens to visual anchors and full tactile sequences, while Contact-gated AVTAG applies a training-only hinge ranking loss to encourage attention to tactile evidence during contact phases.
On which real-world tasks was VT-WAM evaluated? VT-WAM was evaluated on six contact-rich tasks spanning surface-interaction and constrained insertion regimes, including insert plug, swipe card, and insert tube tasks.
What was the main performance improvement over baselines? VT-WAM achieved a 71.67% average success rate across all tasks, outperforming the strongest baseline Fast-WAM by 26.67% absolute improvement.