DeCAL is a physically grounded dexterous vision-language-action model that enables effective utilization of tactile signals for contact-rich manipulation. It uses a Mixture-of-Transformers architecture composed of three collaborative experts for understanding, imagination, and action generation. The three experts perform directional knowledge sharing through joint attention, providing implicit guidance for coherent and fine-grained dexterous manipulation. An adaptive visuo-tactile fusion mechanism dynamically injects tactile information through a contact-aware gating strategy, allowing the model to selectively emphasize tactile cues during physical interactions. Visuo-tactile latent co-imagination jointly models visuo-tactile dynamics through future-oriented latent imagination, equipping the policy with implicit world knowledge.
Extensive real-world experiments evaluate DeCAL on diverse contact-rich dexterous manipulation tasks. DeCAL consistently achieves state-of-the-art performance, outperforming the strongest baseline by more than 15% average success rate. It maintains efficient real-time performance, achieving an average inference latency of 0.27 seconds per action chunk. Ablation studies verify the effectiveness of each proposed component and highlight the promise of effectively leveraging tactile information for dexterous manipulation.
The contributions are as follows:
- DeCAL is a dexterous Vision-Tactile-Language-Action framework that unifies understanding, generation, and action for effective visuo-tactile representation learning.
- Adaptive Visuo-Tactile Fusion and Visuo-Tactile Latent Co-Imagination facilitate contact-rich and fine-grained dexterous manipulation.
- The effectiveness and generalization of the method are demonstrated through a range of real-world experiments.
Robot System Setup
The system consists of a pair of 6-DoF UR5 robotic arms and two 22-DoF SharpaWave five-fingered dexterous hands. Visual observations are captured from two wrist-mounted cameras and one egocentric camera, all using Intel RealSense D435.
Each fingertip is equipped with a high-resolution vision-based tactile sensor developed by Sharpa, enabling fine-grained perception of contact dynamics during manipulation. A built-in camera inside each sensor captures deformation of the elastic surface during physical interaction.
Based on the raw tactile images and visuo-tactile processing algorithms, tactile signals are represented in three complementary forms:
Raw Image
Raw tactile observations are directly captured by the built-in camera inside each visuo-tactile sensor, recording contact patterns and elastic surface deformations.
Net Force
Each fingertip outputs a 6-DoF force representation consisting of three-axis forces and torques. The force signals are estimated from raw tactile observations using a pretrained regression model, enabling force-aware tactile perception.
Deform Map
Each fingertip outputs a deformation depth map represented as a 2D image through a translation model. Each pixel indicates the local deformation depth on the tactile surface, providing fine-grained geometric contact information.
Model Architecture
DeCAL adopts a Mixture-of-Transformers architecture that integrates scene understanding, visuo-tactile dynamics foresight, and action generation within a unified framework. The framework consists of three specialized transformer experts.
Understanding Expert
The understanding expert is built upon Qwen3-VL for its multimodal understanding capability. Given language instructions and multi-view visual observations, the inputs are first encoded into text and visual tokens, which are then processed by transformer blocks to produce contextual embeddings shared with downstream experts through masked self-attention.
The multimodal inputs are extended to tactile observations through a dedicated cross-attention mechanism. Tactile features serve as keys and values, while visual-language features act as queries. The resulting representations are residually fused with the self-attention features, enabling joint reasoning over visual, linguistic, and tactile observations within a unified latent space.
Generation Expert
The Generation Expert explicitly models visuo-tactile dynamics by predicting future visual and tactile observations from historical interactions, forming physically grounded multimodal world representations to guide downstream action generation.
Despite the progress of recent video generation models in visual prediction, directly applying generative modeling to contact-rich manipulation remains challenging because of the strict real-time requirements of high-frequency robot control. DeCAL adopts a parallel decoding strategy for future visuo-tactile generation. This non-autoregressive paradigm improves computational efficiency while remaining effective for modeling future visuo-tactile interactions.
Adaptive Visuo-Tactile Fusion
Tactile Feature Extraction
Multi-finger deform maps are used as tactile inputs because of their rich geometric contact information. A shared ResNet-based tactile encoder extracts tactile features from all fingertips. The extracted features are spatially pooled and projected into tactile tokens, which are subsequently fused through transformer layers to capture cross-finger interactions and global contact patterns.
The tactile encoder produces two complementary representations:
- Local tactile tokens, which preserve fine-grained contact details for visuo-tactile interaction.
- A global tactile token, which summarizes the overall contact state for contact-aware tactile gating.
Adaptive Visuo-Tactile Cross-Attention
To incorporate tactile feedback into visual-language embeddings, a cross-attention mechanism is introduced at each transformer block. The local tactile tokens serve as keys and values, while the prefix projection of the transformer input provides the queries.
The resulting fused output is added to the embeddings from the block’s joint self-attention through a residual connection. This allows the model to integrate tactile cues while preserving the original multimodal context.
Contact-Aware Gating Mechanism
In dexterous manipulation, physical contact is often intermittent and arises only during specific phases of a task rather than being continuously present. Directly injecting tactile signals into the policy at all times can introduce biases in action generation and perturb the distribution of visual attention.
A Contact-Aware Gating Mechanism allows the policy to adaptively modulate the weights of tactile inputs based on the relevance of contact information at each timestep.
The gated output is computed by weighting tactile information according to a learned contact-relevance signal before integrating it into the multimodal representation.
Visuo-Tactile Latent Co-Imagination
The generation expert performs future-oriented latent imagination to jointly model visuo-tactile dynamics. Given historical and current visuo-tactile observations, the model produces predictive latent representations that capture the underlying physical dynamics of the interaction.
Tactile Latent Prediction
6-DoF contact force provides a compact and physically meaningful representation of contact dynamics, capturing both force magnitude and torque information. For tactile latent prediction, the forces from each fingertip are first mapped into embeddings using a dedicated force encoder, which are then processed by the policy to produce a tactile latent.
The resulting latent can subsequently be decoded into representations of different granularities, including raw images, deform maps, and 6-DoF force vectors. These representations capture tactile dynamics from multiple perspectives, supporting physically grounded scene understanding and action generation.
During training, predicted visual latents are directly regressed toward the Cosmos VAE targets encoded from future visual observations. Tactile latents, conditioned on current 6-DoF forces, are indirectly supervised by reconstructing future raw tactile images, deform maps, and 6-DoF forces.
At inference time, the generation expert jointly predicts future visual and tactile latents to provide forward-looking context for action generation. The reconstruction decoders are omitted during inference, avoiding unnecessary pixel-space reconstruction.
Inference Pipeline
During inference, DeCAL embeds multi-view visual observations, language instructions, and tactile inputs into prefix and middle tokens, which are processed by the understanding expert and generation expert.
The prefix tokens produce contextual representations stored in key-value caches, allowing subsequent computations to efficiently attend to all prior context. Middle embeddings, including predicted visuo-tactile foresight, are then integrated with these cached states to form latent representations that capture both task understanding and visuo-tactile dynamics.
Finally, actions are generated iteratively via factorized flow matching, conditioned on the current state and cached context. Running on a single NVIDIA RTX 4090 GPU, DeCAL achieves an average inference latency of 0.27 seconds per action chunk, demonstrating efficient inference for real-time dexterous manipulation.
Supplementary Generation Results
Generated future tactile deformation maps are visualized in the supplementary results. Although the generated tactile observations do not perfectly reconstruct fine-grained deformation details and exhibit some loss of local structure, they successfully preserve contact-related information.
In particular, high-response regions consistently correspond to fingertip-object contact areas, providing informative representations of future contact states. This suggests that the model captures the underlying evolution of tactile interactions, which is sufficient for guiding downstream action generation.
Supplementary Generalization Analysis
The generalization experiments evaluate the models under four types of unseen scenarios:
- Unseen background
- Unseen lighting condition
- Cluttered scene
- Unseen object
The first three settings mainly introduce environment-level perturbations, while the unseen-object setting introduces an object-level distribution shift.
DECO and ViTacFormer benefit from random image augmentations during training, which improves their robustness to visual changes such as background variation, illumination shifts, and moderate scene clutter.
Conclusions
DeCAL is a physically grounded dexterous vision-language-action model that effectively leverages tactile feedback for contact-rich manipulation. By unifying understanding, generation, and action within a collaborative framework, DeCAL enables coherent multimodal reasoning and fine-grained dexterous control.
DeCAL adaptively fuses visuo-tactile information through contact-aware tactile gating and models visuo-tactile dynamics via future-oriented latent imagination, endowing the policy with implicit world knowledge. Extensive experiments on real-world dexterous tasks demonstrate the effectiveness and generalization of the approach.
Limitations
First, the method relies on accurate tactile perception, which can be affected by sensor noise, calibration errors, and model drift. These factors can potentially lead to performance degradation during long-term and high-load operation.
Second, the teleoperation system does not provide fingertip force feedback to the operator. This may result in delayed contact adjustment, suboptimal contact regulation, and limited quality of tactile-aware demonstrations.
Moreover, the current framework does not leverage large-scale visuo-tactile pretraining. Scaling up diverse tactile dexterous data for pretraining could further improve the model’s capability and is an important direction for future research.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (62476011) and the Beijing Natural Science Foundation (L252060).
The authors express their sincere gratitude to Yifan Ye and Yunfan Lou for their insightful discussions and valuable feedback on the methodological design. They are also grateful to Xiansheng Chen for support with the hardware setup.
Frequently Asked Questions
What is DeCAL?
DeCAL is a physically grounded dexterous vision-language-action model that combines visual, language, and tactile information for contact-rich manipulation.
How does DeCAL use tactile information?
It uses deform maps, force representations, and raw tactile observations through adaptive visuo-tactile fusion and contact-aware gating.
What is visuo-tactile latent co-imagination?
It is a future-oriented mechanism that jointly predicts visual and tactile latent representations to capture physical interaction dynamics.
How fast does DeCAL perform inference?
DeCAL achieves an average inference latency of 0.27 seconds per action chunk on a single NVIDIA RTX 4090 GPU.