Dexterous manipulation is a promising direction for enabling robots to perform complex object interactions, but collecting enough high-quality robot demonstrations remains a major bottleneck. For high-degree-of-freedom hands, teleoperation interfaces are particularly costly to design and operate, and the collected motions are tied to a specific robot morphology. Human videos offer a way around this bottleneck: they are easy to collect without a robot or a teleoperation rig and contain rich visual and kinematic information about hand–object interactions. Recent methods exploit this to learn dexterous manipulation from just a single human video.
Introduction
However, these single-video methods mostly convert the video into execution-level guidance, such as reconstructed object trajectories, retargeted hand motions, or contact cues. Such guidance facilitates high-dimensional dexterous policy learning, but also ties the resulting policy to the motion and spatial configuration shown in the video. Even for the same task, these methods generalize poorly to initial object poses, goal poses, and grasps not shown in the video.
Learning a dexterous policy that generalizes beyond the execution shown in the video raises two tightly coupled challenges. The first is task abstraction: deciding what to preserve from the video and what to leave free to vary. Preserving too much reduces the policy to replaying the recorded motion; preserving too little leaves the task underspecified. The second is a hard exploration problem: the more that is left free to vary, the more of the state space the policy must explore. Without execution-level guidance, reinforcement learning (RL) struggles to discover multi-finger, contact-rich interactions in this high-dimensional space, especially in multi-stage tasks.
To address both challenges, we present Dex-One2Many, a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video. The key idea is to extract not the video’s motion but its task structure: a stage-wise scene graph specifying which hand–object and object–object relations must hold, and in what order, while leaving RL to learn how the hand acts. We introduce two techniques.
First, Stage-Wise Scene Graph Abstraction reduces the video to the relations the task requires, such as the hand holding an object or an object resting inside a container, and the order in which they occur. Object poses, goal poses, and grasps remain free to vary. The abstraction splits the video into stages when these relations change and encodes each stage as a graph of predicates over the hand, objects, and their functional parts. Each stage graph describes not one recorded state but a set of states that all preserve the task.
Second, Scene-Graph-Grounded RL uses each stage graph as a generative constraint: it samples reset states that satisfy the stage’s relations, with diverse multi-finger grasps, and keeps only those that pass the simulation checks. One video thus becomes many training states across the whole workspace. The accepted states form the initial-state distribution of a Markov decision process (MDP) whose observations and rewards are derived from the graph. Starting episodes at every stage eases exploration: the policy can practice the last stage before mastering the first, rather than reaching it only by chance. Within each stage, predicate-derived rewards give a dense signal of progress toward satisfying the stage’s relations, without reference to a demonstrated trajectory. The policy therefore learns the task structure rather than the recorded execution.
Trained entirely in simulation, the resulting policy completes the demonstrated task zero-shot on a real multi-fingered hand across initial and goal object poses not shown in the video. We evaluate five tasks spanning direct object manipulation and tool use on a UR3 arm with a 20-DoF Wuji 1 hand. On configurations with initial and goal poses not shown in the video, Dex-One2Many achieves a 60–85% real-world success rate, while the baselines remain at or below 15%.
The main contributions are:
- Stage-Wise Scene Graph Abstraction: We represent a human demonstration as a stage-wise scene graph that retains task-defining structure—the required hand–object and object–object relations and their ordering—while discarding execution-specific motion.
- Scene-Graph-Grounded RL: We use each stage’s scene graph to generate diverse reset states through the task, enabling the policy to explore alternative grasps and motions while grounding its observations and rewards in the graph.
- Zero-shot sim-to-real transfer on unseen configurations: Across five real-world tasks, policies learned from a single human video achieve 75% mean success on configurations not shown in the input video, exceeding the next-best baseline by 71 percentage points.
Stage-Wise Scene Graph Abstraction
We abstract the human video as a sequence of stage-wise scene graphs that preserve which relations must hold and in what order, while allowing object poses, goal poses, grasps, and trajectories to vary. Each graph represents one stage of the task: its nodes identify the hand, task-relevant objects, and their functional parts, while its edges describe the relations characterizing that stage. When the same task is performed from different configurations, the poses and trajectories may differ, but the required relations and their order remain the same. We refer to this abstraction as Stage-Wise Scene Graph Abstraction.
Experiments
We evaluate Dex-One2Many in simulation and on real hardware. The experiments address three questions:
- Does Dex-One2Many generalize to initial and goal configurations not shown in the single human video?
- Does Dex-One2Many apply across diverse dexterous embodiments with different kinematics?
- Which design choices of Dex-One2Many are critical to its performance?
Experimental Setup
Tasks. We evaluate Dex-One2Many on five manipulation tasks spanning direct object manipulation and tool use. The direct-manipulation tasks—Doll, Can, and Stamp—require the robot to grasp an object and move it to a target configuration. The tool-use tasks—Hammer and Sweep—require the robot to grasp and manipulate a tool to interact with another object.
Hardware setup. We conduct all real-world experiments on a 6-DoF UR3 robot arm equipped with a 20-DoF Wuji 1 dexterous hand. Two Intel RealSense D435i RGB-D cameras capture the workspace from complementary viewpoints. We estimate object poses from these observations using FoundationPose++.
Baselines. We compare Dex-One2Many against two prior methods that learn dexterous manipulation from a single human video. These methods differ in what information they extract from the human demonstration and how they use it for policy learning. Do as I Do reconstructs the 4D hand–object interaction and retargets the recovered human hand motion to the robot through trajectory optimization. CHORD uses the demonstrated object-state evolution as an imitation reward and augments it with contact-wrench guidance that encourages the robot to reproduce the demonstrated trajectories.
Generalization Beyond the Human Video
The baselines derive execution-level guidance from the demonstrated hand or object motion, which is tied to the execution observed in the video. In contrast, Dex-One2Many uses stage-wise scene graphs as generative constraints for diverse task-consistent training states, allowing the policy to learn the same task structure across different configurations beyond the demonstrated execution.
Design Choices and Evaluation Protocol
We conduct all ablation studies on the Sweep task in simulation. Success is measured on full-task episodes from empty-hand starts. We study two aspects of Dex-One2Many: the contribution of its learning components and the effect of training scale. For component ablations, we remove one component at a time while keeping all other training settings unchanged. For the training-scale analysis, we vary only the number of parallel environments used for PPO training.
Predicate Vocabulary
The predicate vocabulary covers common relations in tabletop manipulation. Each geometric predicate also has an approach form, denoted by the prefix pre-, which holds while the relation is being approached but not yet established.
Observations
The scene graph identifies the task-relevant entities, their roles, relations, and functional regions. We map the demonstrated hand to the robot end-effector and assign the remaining objects according to their roles in the task. For direct manipulation, the observation contains the end-effector, the manipulated object, and the receptive object. For tool-use manipulation, the held object is instead the tool, and an additional acted-on (interactive) object is introduced.
Object and functional-part nodes provide their local geometry, such as object size, grasp regions, and working surfaces. Functional parts are associated with the hand or another object according to their role in the interaction. We represent a grasp region by its center and bounding-box extents, and a working surface by its center, unit normal vector of its surface, and the extent of the surface.
Each scene-graph edge contributes a relative 6D pose between its connected entities. Importantly, this relative pose is defined according to the predicate encoded by the edge: the predicate determines which geometric features of the two entities are used, following the same feature definitions used for the predicate distance functions. Thus, an edge may relate object centers, functional regions, or other predicate-specific features rather than simply the two object centers. All node, part, and edge features are concatenated to form the policy observation.
Generalization to Unseen Initial Object Positions
To further analyze spatial generalization beyond the human video, we fix the goal configuration and vary the initial object position across the feasible workspace. Initial-position generalization is evaluated across the five tasks.
Diverse Grasp Strategies Across Embodiments and Configurations
The task specification extracted from the human video does not prescribe a specific grasp configuration. Instead, the reset-generation pipeline exposes the policy to diverse multi-finger grasps drawn from the 33-type grasp taxonomy, allowing the grasp strategy to adapt to the robot embodiment and the current object configuration.
Conclusion, Limitations, and Future Work
Although Dex-One2Many substantially improves generalization across unseen configurations and applies to diverse hand morphologies, several limitations remain. First, a sim-to-real gap remains because real-world deployment relies on estimated 6D object poses. Pose-estimation noise and differences between simulated and real object geometry can degrade policy performance, particularly during contact-rich interactions.
Second, the real-to-sim pipeline reconstructs only rigid objects, so articulated objects would require manually built assets. Since a single video may not reveal the underlying articulation structure, such as joint locations, axes, and joint limits, automatic real-to-sim reconstruction remains difficult. Inferring these structural kinematics directly from unconstrained video is an important avenue for future work.
Acknowledgment
This work was partly supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [No. RS-2021-II211343, Artificial Intelligence Graduate School Program (Seoul National University)], the Technology Innovation Program (RS-2025-25453780, Development of a National Humanoid AI Robot Foundation Model for Multi-Task Applications) funded by the Ministry of Trade, Industry & Energy (MOTIE, Korea), and Samsung Electronics Robotics eXperience (RX). We thank Jinu Pahk and Kangin Lee from Tommoro Robotics for providing the Wuji 1 hand used in our real-world experiments.
Frequently Asked Questions
What does Dex-One2Many learn from a human video? It extracts stage-wise task relations and their order, rather than reproducing the demonstrated motion.
How does the method generate training states? It uses each stage’s scene graph to sample candidate reset states, including diverse multi-finger grasps, and retains those that pass simulation checks.
Which tasks are evaluated? The five tasks are Doll, Can, Stamp, Hammer, and Sweep, covering direct object manipulation and tool use.
What are the main limitations? Real-world performance can be affected by object-pose estimation and simulation-to-reality geometry differences. The pipeline also reconstructs only rigid objects.