A new framework called TacForeSight lets robots predict upcoming tactile sensations from wrist force and torque readings, enabling them to handle contact-rich tasks with much greater robustness. On five challenging manipulation tasks, it achieved a 79% average completion rate and 86.7% success under unexpected disturbances, significantly outperforming existing methods.
What the Researchers Built
TacForeSight is a two-part predictive system for contact-rich manipulation. The first component, TacForceWM, is a tactile world model that takes high-frequency wrist force/torque readings (120 Hz) and recent tactile sensor data (30 Hz) to predict short-horizon tactile latent states — essentially foretelling what the robot’s fingers will feel in the next few moments. The second component is a Predictive Tactile-Conditioned Policy that uses those predicted tactile latents as anticipatory priors to guide action sequences.
The researchers mounted two Xense tactile sensors on the fingertips of a Robotiq 2F-85 gripper attached to a 7-DoF xArm7 robot. A UFactory 6-axis force/torque sensor at the wrist captures global interaction forces at four times the tactile capture rate. This setup reflects how humans use load cues to anticipate contact states before explicit tactile feedback arrives. The system was tested on five representative tasks, including peg insertion, surface following, and cable routing, under both nominal conditions and three types of in-process perturbations (height, angle, and pose changes).
Key Results
TacForeSight was compared to several baselines: vision-only policies, reactive tactile feedback controllers, and prior tactile prediction methods. The evaluation used a completion score (percentage of successful task executions) across multiple trials.
| Condition | TacForeSight (proposed) | Best Baseline |
|---|---|---|
| Nominal (5 tasks average) | 79.0% | ~55% (estimated from text) |
| Height perturbation | 90.0% | ~60% |
| Angle perturbation | 85.0% | ~50% |
| Pose perturbation | 85.0% | ~45% |
| All perturbations average | 86.7% | ~52% |
Under nominal conditions, TacForeSight outperformed all baselines by a wide margin, particularly on tasks involving multi-stage contact transitions or precise force regulation. When disturbances were introduced mid-execution — for example, suddenly raising the table height, tilting the surface, or shifting the object — baseline methods frequently lost contact or caused slippage. TacForeSight recovered quickly, re-establishing stable contact and continuing the task without losing progress.
Ablation studies confirmed that each component contributes meaningfully. Removing the force conditioning from the world model dropped average performance by over 15%, while removing the predictive tactile-conditioned policy reduced perturbation recovery by 20%.
How It Works
TacForeSight operates in two phases: offline pretraining and online deployment. During pretraining, the tactile world model (TacForceWM) learns to map sequences of force/torque readings and tactile observations to future tactile latent states. The tactile tokenizer uses a hybrid CNN-Transformer architecture to compress the dense 3D displacement maps from each fingertip into compact frame-level tokens. These tokens are augmented with learnable spatial positional embeddings and finger-specific identity embeddings so the model can distinguish between the two fingers and their local contact geometry.
The force encoder processes the high-frequency wrist wrench signal (force and torque along three axes each) and conditions the latent dynamics predictor. This predictor learns to forecast the evolution of tactile latents over a short horizon (typically 2–3 timesteps ahead) given the current tactile state and the force trajectory. The key insight is that global force/torque changes at the wrist precede local tactile changes at the fingertips by several milliseconds — by the time tactile sensors register a change, the interaction has already shifted. Predicting tactile latents from force allows the system to act proactively.
In online deployment, the Predictive Tactile-Conditioned Policy receives multimodal observations: camera images (30 Hz), joint positions, current tactile latents, and the predicted future tactile latents from TacForceWM. A cross-attention mechanism models the evolution from current to future tactile representations, producing an anticipatory contact prior. A tactile-guided gate then adaptively fuses this prior with visual features, controlling how much the policy relies on tactile information versus vision depending on the contact state. Finally, a conditional flow-matching action head generates smooth, contact-aware action sequences.
The entire pipeline runs at 30 Hz, meeting real-time requirements for manipulation. The compact latent space (64 dimensions per tactile frame) keeps computation light enough for onboard inference.
Why This Matters for Robotics
Contact-rich manipulation — tasks like inserting a peg into a hole, wiping a surface, or following a contour — remains one of the hardest challenges in robotics. Traditional vision-based control struggles with occlusions and low-fidelity depth data at close range. Reactive tactile feedback, while helpful, always lags behind the actual physical interaction. By the time the robot feels a slip, the object may already be misaligned.
TacForeSight flips this paradigm. Instead of reacting to tactile events, the robot predicts them from force signals that arrive earlier. This proactive capability makes manipulation more robust to unexpected disturbances, which are common in real-world environments: a workbench gets bumped, a part shifts in the gripper, or the lighting changes and vision degrades.
For buyers of used industrial robots or warehouse robots looking to automate assembly or packaging tasks, this approach could mean fewer dropped parts, higher throughput, and less need for structured fixtures. Companies deploying collaborative robots for precision assembly could benefit from the enhanced robustness to misalignment and environmental variations.
The modular design also means the force-conditioned world model could be adapted to other sensor configurations or robot morphologies, though the current experiments use a specific dual-finger gripper setup.
Limitations and Open Questions
TacForeSight was tested only on a single robot platform with a parallel-jaw gripper. The world model assumes a fixed sensor layout and might not transfer directly to different grippers, multiple fingers, or soft hands without retraining. The short prediction horizon (2–3 timesteps) is sufficient for local adjustments but does not provide long-horizon foresight for complex sequences.
The force-torque sensor used (UFactory 6-axis) is a high-quality industrial sensor; lower-cost or noisier sensors may degrade prediction quality. The method also requires both tactile and force sensing, which adds hardware cost and complexity not all setups have.
Generalization across objects with very different compliance (e.g., soft foam vs. rigid metal) was not extensively studied, and the tactile sensors used (Xense) are displacement-based, not force-sensitive. Whether the same approach works with other tactile modalities like GelSight or taxels is an open question.
Frequently Asked Questions
What hardware does TacForeSight require? It requires a robot arm with a wrist-mounted force/torque sensor (at least 120 Hz) and tactile sensors on each fingertip, plus a wrist camera. The researchers used a 7-DoF xArm7, Robotiq gripper, RealSense D435, UFactory F/T sensor, and Xense tactile sensors.
How fast does the system run? The full prediction and control pipeline runs at 30 Hz, matching the camera and tactile sensor frame rate, which is sufficient for real-time manipulation.
Can this work without force sensors? Force conditioning is central to the method — removing it dropped performance by over 15%. A low-cost force sensor might work, but the quality of high-frequency force signals is critical for accurate tactile prediction.
Does it work on soft or deformable objects? The paper tested on rigid and semi-rigid objects (peg, cable, surface). Soft or highly deformable objects may require different tactile sensing and may not follow the same force-to-tactile dynamics.
Conclusion
TacForeSight introduces a practical way to give robots predictive touch awareness by linking high-bandwidth wrist force signals to future tactile states. With strong results in both nominal and perturbed conditions, it offers a clear path toward more reliable contact-rich manipulation in real-world settings.