World models enable robots to learn predictive representations of how the environment evolves, and recent World Action Models (WAMs) combine such predictive learning with action generation for robot control. However, most WAMs still model the future primarily through visual state transitions. Visual prediction captures scene layout, object appearance, and visible motion, but physical manipulation also depends on local mechanics that are only indirectly expressed in images. For contact-rich tasks, learning how the scene looks is therefore not sufficient; the model should also capture how tactile signals such as force and deformation change during interaction.
Prior visuo-tactile methods have used touch for observation fusion, representation learning, policy refinement, and future tactile prediction. These works show that touch can improve manipulation and that future tactile prediction can be useful. Building on this direction, we focus on a complementary requirement for tactile WAMs: future tactile states should provide physically informative supervision while action generation remains restricted to information available at deployment. This lets tactile futures enrich world-model training without serving as direct action inputs.
Contributions
- We introduce TacWAM, which extends visual WAM training with recent tactile history and future tactile prediction, enabling the model to learn from changes in touch beyond the current tactile observation.
- We develop a Spatially Aligned Fusion Tactile Encoder that maps tactile appearance, force, and deformation into a shared latent representation while preserving local contact patterns and global force/torque information.
- We propose Anchor-Guided Tri-Modal Attention, which prevents the action branch from accessing future visual or tactile targets during training, ensuring that action generation uses only information available at deployment.
- We evaluate TacWAM on four real-world contact-rich manipulation tasks. TacWAM consistently outperforms representative vision-only and visuo-tactile baselines, while staged ablations support the importance of tactile history and restricted future-target visibility in the complete framework.
Visuo-Tactile and Predictive Tactile Learning
TacWAM builds on the broader visuo-tactile learning direction rather than treating tactile prediction itself as the primary novelty. It focuses on how tactile futures are represented and how they are used inside WAM training: the SAF Encoder and tactile history support prediction of force and deformation changes, while AGT Attention keeps future sensory targets from becoming direct inputs to action generation.
TacWAM Overview
TacWAM is built around a single tri-modal WAM generator with visual, tactile, and action streams. On top of this generator, TacWAM introduces three tactile-specific designs. First, the SAF Tactile Encoder maps synchronized tactile appearance, force, and deformation-flow signals into a shared tactile latent space, which defines the future tactile prediction target. Second, the tactile history encoder summarizes recent tactile changes before the current chunk and conditions the tactile prediction branch on contact-phase context. Third, AGT Attention specifies the information topology among visual, tactile, and action streams, preventing future sensory prediction targets from being directly accessed by action tokens.
The three streams are jointly implemented in a Mixture-of-Transformers backbone, where visual, tactile, and action streams are processed by modality-specific experts and communicate through masked mixed self-attention. AGT governs this mixed self-attention among visual, tactile, and action tokens; task language and current proprioception are supplied separately as deployment-available context through expert cross-attention. Under the finalized AGT mask, future tactile prediction is a parallel predictive objective rather than an action-conditioned consequence model: the action stream uses deployment-available visual and tactile anchors, task/proprioceptive context, and action-token interactions, while future tactile states provide predictive supervision for force and deformation changes during training.
Spatially Aligned Fusion (SAF) Tactile Encoder
TacWAM defines a tactile latent prediction space using a tactile encoder and a tactile reconstructor, as illustrated in the accompanying figure. Each tactile observation contains synchronized signals from the two gripper-side tactile sensors, where the inputs are the rectified tactile image, the dense local force field, and the mesh deformation flow. These three inputs are spatially registered on the tactile sensor surface and are fused into a step-level tactile representation.
The resultant wrench is therefore treated as global mechanics supervision rather than a fourth spatial input. TacWAM predicts future tactile latent states instead of directly generating heterogeneous tactile sensor streams. The reconstruction objectives encourage the latent state to preserve local force, deformation, and bilateral wrench information.
Tactile History-Modulated Prediction
Single-frame touch is often ambiguous: the same force magnitude can correspond to stable contact, growing pressure, slip recovery, or impending object damage depending on the recent interaction trajectory. TacWAM therefore summarizes recent tactile history before the current chunk.
The output is a compact chunk-level tactile context. It is not inserted as additional memory tokens. Instead, it modulates the tactile expert through adaptive normalization, helping future tactile prediction disambiguate the current interaction phase and tactile evolution trend. The history context is not a direct privileged input to the action stream.
This design changes the role of tactile prediction. TacWAM does not attach tactile forecasting as an isolated post-hoc head. Instead, future tactile states are integrated into the multimodal generative objective, encouraging the jointly trained WAM to encode contact-relevant tactile evolution beyond visual supervision alone.
Experiments
We evaluate TacWAM through the following research questions:
- RQ1: How does TacWAM compare with vision-only and visuo-tactile baselines on contact-rich manipulation?
- RQ2: Does tactile history improve future force prediction and phase-dependent contact manipulation within TacWAM?
- RQ3: How do AGT information constraints affect deployment performance compared with relaxed attention masks?
Frequently Asked Questions
What is the core difference between TacWAM and existing World Action Models?
TacWAM extends visual WAM training by adding recent tactile history and future tactile prediction, allowing the model to learn from changes in force and deformation rather than visual state transitions alone.
How does the Spatially Aligned Fusion Tactile Encoder work?
It maps tactile appearance, force, and deformation-flow signals into a shared latent space while preserving local contact patterns and global force/torque information through reconstruction objectives.
Why does TacWAM prevent the action branch from accessing future sensory targets?
To ensure that action generation during deployment uses only information available at that time, keeping future tactile states as predictive supervision rather than direct action inputs.
What tasks were used to evaluate TacWAM?
TacWAM was evaluated on four real-world contact-rich manipulation tasks, consistently outperforming vision-only and visuo-tactile baselines across all tasks.