Introduction
Importantly, these inner policies assist data collection; they are not components of the final autonomous controller. The resulting demonstrations are then used to train a separate outer visuomotor policy that controls both the arm and hand at deployment. To make this high-dimensional action space easier to learn, a hand-action variational autoencoder (H-VAE) provides compact hand targets while retaining arm commands directly in joint space. We evaluate the resulting collection-to-autonomy pipeline from demonstration collection and autonomous policy learning through online execution, contact behaviour, and reuse of hand skills across tasks.
Concretely, the technical contributions are:
- Copilot-assisted dexterous demonstration collection. A reversible clutch allows an operator to regulate state-conditioned hand skills while controlling the arm, with automatic skill selection enabling different hand behaviours to be reused across task stages.
- Nested collection and autonomous deployment. Learned inner hand policies, dynamically allocated by a vision-language model, assist complete-task demonstration collection, while a separate outer visuomotor policy learns from the resulting demonstrations and executes independently at deployment.
- Real-world collection-to-autonomy evaluation. Across six dexterous manipulation tasks, demonstration reliability and efficiency, downstream autonomous policy learning, and the effects of compact hand-action representations and online contact-aware execution are evaluated.
NestDex: Nested Dexterous Policies
NestDex separates assisted demonstration collection from autonomous task execution. Retargeted hand-skill demonstrations train proprioceptive inner policies, which assist complete-task teleoperation. A variational autoencoder (VAE) for hand poses, called H-VAE, then encodes the collected hand commands for training an outer visuomotor policy that controls the complete task.
Implementation
The experiments use a leader-follower platform. Each leader has an arm and a clutch, while each follower has an arm, a dexterous hand, and a wrist camera. Leader-arm joint positions map directly to follower-arm commands. The leader clutch regulates inner-policy execution on the follower hand, and each arm-hand pair executes its inner policy independently.
Reliable and Efficient Demonstration Collection
NestDex was compared with AnyTeleop across six dexterous manipulation tasks, collecting twenty demonstrations per task with each system. Success rate and time per successful demonstration were reported under the defined metrics. “N/A” indicates that no successful demonstration was obtained.
The same operator performed both conditions after becoming familiar with each interface. Objects were reset to the same nominal starting configurations between attempts.
From Copilot to Autonomy
Outer policies trained on copilot-collected demonstrations achieve effective success rates across all evaluated tasks, confirming the utility of these demonstrations for downstream policy learning.
With direct hand actions, policies trained on copilot-collected demonstrations outperform those trained on AnyTeleop demonstrations on every task for which both sources provide training data. For Tongs Transfer, AnyTeleop does not provide successful demonstrations for outer-policy training, whereas copilot-collected demonstrations support successful autonomous task execution.
Online Execution and Contact Behaviour
The same inner policy was tested on a water-bottle grasp under three execution conditions:
- Fixed-command replay: Repeats the joint-command trajectory saved from one successful rollout.
- Closed-loop, no ensemble: Queries the policy from the latest proprioceptive history but executes the first action of each new chunk.
- Closed-loop, temporal ensemble: Applies temporal averaging of overlapping predictions from the same policy.
For a balanced comparison, the first ten trials with an adjudicated outcome from each condition were used, giving thirty trials in total.
Skill Selection and Reuse Across Task Stages
Inner-policy selections were qualitatively examined during representative Toast Preparation and Binder Filing demonstrations.
In both tasks, the selected skill follows the current stage, and previously used skills are reactivated when needed. Toast Preparation selects Tongs Grasp, Button Press, and Plate Grasp before returning to Tongs Grasp. Binder Filing returns to Paper Pinch after Hole-Punch Press, then selects Binder Closure.
These sequences show that the selector reuses a compact set of hand skills across task stages.
Frequently Asked Questions
What is the role of the inner policies in NestDex? Inner policies assist demonstration collection and are not components of the final autonomous controller.
What does the outer visuomotor policy control? The outer policy controls the complete task, including both the robot arm and hand, during deployment.
How does the H-VAE support policy learning? The H-VAE encodes collected hand commands into compact hand targets while arm commands remain directly in joint space.
How are hand skills reused across task stages? Automatic skill selection follows the current task stage and reactivates previously used skills when needed.