Mana is a complete system that trains robot hands to manipulate real-world articulated tools like syringes, scissors, and pliers entirely from simulation data. By combining a novel keyframe-based task planner with a diffusion policy and custom compliant fingertips, Mana achieves zero-shot sim-to-real transfer for dexterous tool use with millimeter-level precision.
What the Researchers Built
The team from Stanford and the University of Washington developed Mana, a data-generation and training pipeline that enables a dexterous robot hand to grasp and actuate common articulated tools — objects with moving parts like syringes, scissors, spray bottles, and clippers. Unlike previous work that focused on rigid objects or simple grippers, Mana tackles the challenge of in-hand manipulation where the robot must transition between multiple grasp configurations to operate the tool.
Mana has three core components: a keyframe planner that generates dense grasp states across relevant tool configurations, a trajectory generator that connects those keyframes into motion sequences, and a visuomotor policy trained entirely on simulation data. The policy uses a diffusion model conditioned on point clouds from a single RGB-D camera. To bridge the sim-to-real gap, the researchers designed custom flattened silicone fingertips that provide stable contact on thin tool parts. The hardware setup uses a 7-DoF xArm7 arm and a 16-DoF Allegro hand, with perception running at 10 Hz on two RTX 4090 GPUs.
Key Results
Mana was evaluated on six diverse articulated tools — syringe, scissors, spray bottle, pliers, clipper, and tweezers — across a total of 120 real-world trials. The system achieved an overall success rate of 86.7%, with particularly strong results on the syringe (95%), clipper (90%), and tweezers (90%). The scissors and spray bottle proved more challenging, at 80% and 75% success respectively.
| Tool | Success Rate (Real-World) |
|---|---|
| Syringe | 95% |
| Clipper | 90% |
| Tweezers | 90% |
| Pliers | 85% |
| Scissors | 80% |
| Spray Bottle | 75% |
Ablation studies confirmed the importance of each component. Removing force-domain randomization dropped success from 86.7% to 61.7%. Replacing the custom compliant fingertips with standard rigid ones reduced performance to 73.3%. The point-cloud randomization for sim-to-real transfer contributed a 10% improvement. The system also maintained robustness under camera viewpoint changes and varied lighting conditions, though performance degraded when the object point cloud was significantly occluded.
How It Works
Mana decomposes articulated tool manipulation into three phases: pre-grasp, grasp, and in-hand actuation. Each phase has distinct physical requirements. The keyframe planner first samples dense grasp states across the tool's functional contact regions, generating thousands of candidate poses for each phase. This density is critical because a 1-2 mm displacement in contact location can cause tool ejection during forceful actuation.
The trajectory generator connects these keyframes using a reinforcement learning policy trained in IsaacLab with PPO. The RL policy uses privileged state information — hand and tool poses, velocities, contact forces, and goal configurations — while the teacher-student framework distills this into a vision-based policy that only sees point clouds. The reward function includes dense terms for tracking desired states plus regularization penalties for excessive force.
For real-world deployment, the system uses a diffusion-based visuomotor policy with a transformer architecture. The encoder processes point clouds through a PointNet++ backbone to produce 128-D tokens, which condition a 6-layer diffusion transformer that predicts the next action. During training, the team applied aggressive domain randomization to force-related parameters — PD gains, mass, friction — because accurate force simulation was found to be the most critical factor for sim-to-real transfer. They also used Implicit Euler integration with increased position and velocity iterations to handle the high forces on small (0.5-1 cm) tool parts without penetration.
Why This Matters for Robotics
Mana addresses one of the most stubborn bottlenecks in robotic manipulation: handling tools with moving parts. Most commercial industrial robots are limited to pick-and-place operations with rigid objects, and even advanced humanoid robots struggle with tasks that require multi-step in-hand manipulation. By showing that a simulation-trained policy can zero-shot transfer to real-world use of syringes, scissors, and clippers, Mana opens the door to automating many tasks that currently require human dexterity.
This capability has direct applications in laboratory automation (operating pipettes and syringes), medical device handling, and warehouse packaging — areas where used cobots for sale could be upgraded with dexterous hands and this type of policy. The key insight that force randomization and custom compliant fingertips are essential for success also provides practical guidance for anyone building manipulation systems for thin, high-force tools.
Limitations and Open Questions
Mana's current implementation has several constraints. The system requires a controlled camera viewpoint and struggles with significant point cloud occlusion. The custom fingertips, while effective, are specialized for thin tools and may perform poorly on objects with very different geometries. The 10 Hz inference rate is adequate for quasi-static manipulation but would need improvement for dynamic tasks. The paper also does not address tool re-grasping or recovery from failure — if the robot drops the tool, it cannot retry without human intervention. Finally, the six tools tested, while diverse, represent only a small fraction of real-world articulated tools; scaling to a wider range of shapes and mechanisms remains an open challenge.
Frequently Asked Questions
What is the Mana system? Mana is a complete data-generation and training pipeline that enables a dexterous robot hand to manipulate articulated tools like syringes and scissors using only simulation data, with zero retraining needed for real-world deployment.
What hardware does Mana use? The system uses a 7-DoF xArm7 robot arm with a 16-DoF Allegro hand fitted with custom flattened silicone fingertips, and a single Intel RealSense D435 RGB-D camera for perception.
How does Mana achieve sim-to-real transfer? Through aggressive domain randomization of force parameters (PD gains, mass, friction), point-cloud noise perturbations, and custom compliant fingertips that tolerate small position errors and provide stable contact on thin surfaces.
What tools can Mana manipulate? The system was successfully tested on six tools: syringe, scissors, spray bottle, pliers, clipper, and tweezers, achieving an overall 86.7% success rate across 120 real-world trials.
Conclusion
Mana demonstrates that dexterous manipulation of articulated tools can be learned entirely in simulation and transferred to the real world with high reliability. By combining a principled keyframe planner, force-aware domain randomization, and custom hardware, the system achieves a new level of capability for robot hand manipulation.