Recent work has explored using humanoid-style retargeting pipelines for dexterous hands, but results have largely been confined to simulation or open-loop replays of simulation motions. One hypothesis for this limited real-world success is that simple kinematic retargeting, while closely matching the human hand pose, ignores the hand-object interaction and produces physically implausible trajectories that are a poor reference for downstream RL. This raises a natural question: does preserving interactions during retargeting improve RL for contact-rich dexterous manipulation?
Our contributions are as follows:
- We propose a general and minimalist framework for learning dexterous manipulation from a single human demonstration. We release the code and data for the community to reproduce the results and build upon our work.
- We apply a simple yet effective data augmentation strategy that exposes the RL policy to diverse trajectories for improved robustness and generalization.
- We evaluate our method on challenging contact-rich dexterous manipulation tasks and demonstrate its superior performance over existing methods, especially the value of interaction-preserving motion retargeting.
Retargeting-Guided Reinforcement Learning Framework
The proposed framework, which we call Regrind, consists of three stages. First, a single human demonstration is captured using motion capture gloves and an object tracker. Second, the human hand motion is retargeted to the robot hand while preserving hand-object interactions through a physics-based optimization. Third, a reinforcement learning policy is trained to track the retargeted trajectory, with data augmentation applied during training to improve robustness.
Interaction-Preserving Motion Retargeting
The key innovation is the interaction-preserving retargeting step. Rather than simply mapping human joint angles to robot joint angles, we optimize the robot trajectory to maintain the same spatial relationship between the hand and object as observed in the human demonstration. This ensures that contact-rich interactions are preserved, yielding physically plausible reference trajectories for downstream RL training.
The retargeting objective balances three terms: a kinematic term that matches the human hand pose, an interaction term that preserves hand-object spatial relationships, and a regularization term that keeps the robot within joint limits and avoids self-collisions.
Data Augmentation for Robust Policy Learning
During RL training, we apply trajectory-level data augmentation to expose the policy to diverse initial states and object configurations. This includes randomizing the initial object pose, adding noise to the reference trajectory, and perturbing the robot's initial joint positions. This simple augmentation strategy significantly improves the policy's ability to generalize to different starting conditions and real-world dynamics.
Experimental Evaluation
We design experiments to answer the following key questions:
- Does interaction-preserving retargeting improve retargeting fidelity?
- Does preserving interactions during retargeting improve downstream RL performance?
- Are the policies trained with Regrind able to transfer to real-world dynamics?
- Can the policies replicate the demonstration from different initial states?
Experiment Setup
Evaluation Metrics. We use the following two metrics to evaluate policy performance:
- Object Tracking Error (Err.): The average distance between current object keypoint positions and target object keypoint positions.
- Success Rate (SR): The ratio of successful trials over total number of trials.
Baselines. We compare our method with two representative baselines for learning dexterous manipulation from human demonstrations:
- SPIDER, a physics-based retargeting method that generates dynamically feasible robot trajectories. The retargeted trajectories can be executed as open-loop policies or used as reference motions for downstream RL.
- DexMachina, which trains RL policies to track kinematic retargeted robot trajectories, but without considering robot-object interaction semantics in the retargeting process.
In addition, we compare with a simple baseline that retargets human motion to the robot with differential inverse kinematics (using Mink) and then trains an RL policy with the same settings as our method, named Mink IK + RL.
Results
Our method consistently outperforms all baselines across multiple contact-rich dexterous manipulation tasks. The interaction-preserving retargeting leads to significantly lower object tracking errors and higher success rates compared to kinematic-only retargeting approaches. Policies trained with Regrind also demonstrate successful sim-to-real transfer, maintaining high success rates when deployed on real hardware.
Limitations
We are optimistic about retargeting-based learning for dexterous manipulation, as we have demonstrated promising results on challenging tasks and gained a better understanding of the challenges of sim-to-real transfer in contact-rich manipulation. At the same time, several limitations remain. First, our method relies on motion capture to obtain the object state at deployment time. Distilling the state-based RL policy into a vision-based policy is a natural next step to make the system more applicable to in-the-wild scenarios. Second, our pipeline still requires careful system identification to enable sim-to-real transfer; a promising direction is to adapt the policy at test time using context inferred from interactions with the real world.
This project was supported in part by the Department of the Navy, Office of Naval Research under ONR award number N00014-25-1-2086. Model training was performed using the Unicorn shared computing cluster at Cornell University. Yunhai Feng would like to thank the PoRTaL Lab for support with motion capture prototyping, Paige Yun for help with the motion capture infrastructure, and Samuel Jin and other members at the Praxis Lab for helpful technical discussion.
Frequently Asked Questions
What makes interaction-preserving retargeting different from standard kinematic retargeting? Standard kinematic retargeting only matches joint angles, while interaction-preserving retargeting also maintains the spatial relationship between the hand and object, ensuring contact-rich interactions from the human demonstration are preserved.
How many human demonstrations are needed to train a policy? The framework requires only a single human demonstration to train a dexterous manipulation policy, making it highly data-efficient.
Can the learned policies transfer to real robot hardware? Yes, the paper demonstrates successful sim-to-real transfer of policies trained with Regrind, showing that the interaction-preserving retargeting produces physically plausible trajectories suitable for real-world deployment.
What types of dexterous manipulation tasks can this method handle? The method is evaluated on challenging contact-rich tasks that require precise finger coordination and force control, such as object reorientation and in-hand manipulation.