Hardware Design
Recorded Signals
During data collection, the Quest 2 controller provides the wrist’s 6-DoF trajectory, from which end-effector motion targets are constructed. The fisheye camera captures hand–object interactions and surrounding task context, while the hand controller records both the commanded 20-DoF joint targets and the resulting joint states. Each demonstration therefore carries the actions actually sent to the hand, rather than actions inferred from measured states, satisfying the Action Data requirement. Rigid connections keep the controller-to-hand and camera-to-hand transforms fixed, so the tracked motion and visual observations remain expressed relative to the same hand frame.
Ergonomic Design
The complete hand-held assembly weighs the total weight is not specified in the provided text kg, of which the non-hand hardware contributes the non-hand hardware weight is not specified in the provided text kg. To support this load while leaving the thumb and index finger free to operate the controls, the Quest 2 battery cover is extended into a collection handle that connects the controller to the transferable module. A strap across the back of the operator’s hand carries part of the load, reducing sustained grip force.
The initial design placed the controller behind the robot hand, offsetting the operator’s grip from the assembly’s center of mass and creating a long moment arm. The final design positions the controller above the hand so that the grip lies near the center of mass. This shortens the moment arm and reduces the wrist torque the operator must counteract.
Dexterous Control via Parameterized Skills
Control via skills is an effective way to operate high-dimensional robotic systems. For example, a humanoid can be commanded through a desired velocity rather than by specifying the motion of every joint. The same principle is applied to dexterous hands by separating the operator’s intent from the finger motions required to realize it. A compact set of skills is designed to work across a wide range of dexterous tasks. The skill interface is parameterized instead of discrete, allowing more holistic control. For instance, the user controls grasp strength by using the joystick to incrementally close or open the grasp, instead of snapping to a single fixed grasp.
Together, the skills keep all 20 joints of the hand available during collection without asking the operator to command each one, satisfying the DoF requirement that separates a dexterous interface from a simpler gripper. The interface uses a Meta Quest 2 controller, which provides two buttons, two continuous triggers, and a two-axis joystick. These inputs allow the operator to select and modulate the skills.
Thumb Control
The two joystick axes control thumb flexion and extension along one axis and abduction and adduction along the other, following HATO. This allows the operator to position the thumb independently for different grasp geometries.
Power Grasp
One joystick axis controls the coordinated closure of the four non-thumb fingers. Moving the joystick incrementally loosens or tightens the grasp, while releasing it holds the current target. Unlike continuous trigger control, this persistent command does not require the operator to maintain pressure throughout a grasp. It is primarily used to wrap the hand around larger objects.
Pinch Grasp
The same incremental command controls only the index finger, which closes against the independently positioned thumb. This skill supports small objects and grasps that require precise fingertip contact.
Two-Finger Grasp
The index and middle fingers open or close together while the remaining fingers retain their targets. This skill supports multi-stage behaviors such as multi-object grasping and two-finger grasp-and-pull motions, without requiring the operator to coordinate the two fingers individually.
In-Hand Rotation
A button invokes a learned closed-loop policy that generates the repeated finger gaiting required to rotate a grasped tool in hand. The policy controls the fingers while the operator continues to control the global motion of the hand.
Learning the In-Hand Rotation Skill
The in-hand rotation skill is learned in simulation. This relieves the operator of difficult low-level in-hand manipulation, especially without finger-level haptic feedback. A closed-loop screwing policy is trained in simulation with PPO in Isaac Lab.
The task is modeled as rotating a cylindrical handle on a revolute joint. In the real world, the operator can make wrist adjustments, but the simulation environment is simplified so the hand base is fixed and the policy commands fingers only. The policy learns delta actions over 12 actuated joints, with the ring and pinky fingers fixed. It observes a three-step history, including measured joint positions, commanded targets, and the previous action. A privileged critic is learned with access to the handle’s rotation angle.
Experiments
The evaluation addresses three research questions:
- Q1: Do the hand commands recorded by DexPort enable policies that succeed on diverse dexterous tasks?
- Q2: Does co-training with demonstrations collected in unstructured environments improve robustness to unseen objects and cluttered scenes?
- Q3: How does DexPort’s collection throughput compare with teleoperation and human-hand demonstrations?
Experimental Setup
Robot Platform
Policies are deployed on Tianji Marvin, a bimanual robot with two 7-DoF arms. The hand–camera modules used for collection mount directly to its wrists, so the hand embodiment and camera-to-hand extrinsics are identical across collection and deployment. Hand targets therefore need no retargeting; the only correction at deployment is the fixed controller-to-end-effector transform.
The system’s modularity is demonstrated with different hands: the Wuji 1 hand for Pick-and-Place and Screwing, and the Wuji 2 hand for Drawer Stowing and Bimanual Barcode Scanning.
Tasks
Four tasks evaluate complementary forms of dexterity:
- Vegetable Pick-and-Place: The robot sequentially picks and places two vegetables into a plate. Variations in object size and shape require the policy to select between pinch and power grasps without explicit grasp labels.
- Screwing: The robot maintains a stable grasp while rotating a screwdriver in hand, testing sustained contact and repeated finger gaiting.
- Drawer Stowing: Without releasing a grasped object, the robot opens a drawer with the same hand, places the object inside, and closes the drawer. This task couples object retention with articulated-object manipulation.
- Bimanual Barcode Scanning: One hand grasps and presents an object while the other positions and operates a barcode scanner. Success requires coordinated bimanual positioning of the object and scanner.
Observations
Policies observe the fisheye view from the hand–camera module. For drawer stowing, a third-person camera is added because the grasped object occludes much of the scene while the drawer is pulled open. The operator is occasionally visible in recorded frames, and no post-processing is applied.
Datasets
A total of 1,605 laboratory demonstrations are collected across the four tasks: 700 for vegetable pick-and-place, 200 for screwdriver manipulation, 380 for drawer stowing, and 325 for bimanual barcode scanning. An additional 1,000 pick-and-place demonstrations are collected in unstructured apartment and office environments. The co-trained policies use both the laboratory and in-the-wild pick-and-place datasets.
Training Variants and Baselines
To answer Q1, policies trained on the commanded hand targets recorded by DexPort are compared against a state-as-action baseline that instead uses measured hand states as action labels. To answer Q2, the laboratory-only pick-and-place policy is compared with two co-trained variants. To answer Q3, the number of trajectories collected per 30 minutes is compared across DexPort, teleoperation, and human-hand demonstrations.
Evaluation Protocol
Each policy is evaluated for 15 episodes per task or evaluation condition. For vegetable pick-and-place, testing covers seen objects, unseen objects, and scenes with distractors. Partial success denotes placing one of the two objects; full success denotes placing both. For drawer stowing and bimanual scanning, the reported results cover completion of the task phases. For screwdriver manipulation, the reported measures are average hold duration in seconds and rotations per minute.
Policy Learning Across Dexterous Tasks (Q1)
Imitation learning results cover all four tasks. Policies trained on DexPort demonstrations acquire the target behaviors: they select between pinch and power grasps without grasp labels, hold an object while pulling open a drawer with the same hand, gait the fingers to rotate a screwdriver in hand, and coordinate two arms to present an object to a scanner.
Data Collection Throughput (Q3)
Throughput is measured as the number of trajectories collected during 30 minutes of continuous operation on the unimanual pick-and-place task, comparing DexPort against bare-hand human demonstration and teleoperation. For Glove Teleoperation, the operator uses the Wuji Glove for hand tracking and the VR controller for the end-effector. For Skill Teleoperation, the operator uses the VR controller for end-effector control and the same hand skills as DexPort. The operator is expert-level for each style of data collection.
Conclusions
Limitations
Despite its ergonomic design, the device’s weight limits uninterrupted use and rotation-heavy motions. Because the weight of the hand is the primary bottleneck, future hardware developments for lighter and smaller hands may address this. The skills also cover only part of the space of dexterous behaviors. Future work may explore extending them with additional skills learned in simulation, such as other in-hand manipulation skills, learned grasps, and more.
Acknowledgment
The authors thank Keyshawn King and Volodymyr Behma for data collection and support, and Andrew Wright for initial hardware designs. They also thank Tingfan Wu, Anas Houssaini, Jensen Gao, and the FAR team for feedback and support.
Hardware and Data Collection
Hardware
The hand–camera module pairs a Wuji hand with a USB board camera with a 200° fisheye lens. On the Wuji 1, hand targets pass through a 5 Hz low-pass filter and are interpolated by the hand firmware at 1 kHz, with a per-joint torque limit of 0.5 N·m.
Recording and Synchronization
A single recording loop runs at 30 Hz. The camera, hand, and Quest controller are each read in a background thread. Because the camera and hand use the same 30 Hz loop during collection and deployment, their latencies are expected to roughly match across the two settings. The streams are therefore not latency-measured or temporally aligned in post-processing, as UMI does. At deployment, asynchronous inference discards actions whose time has already passed.
Deployment
Control
The policy runs at 30 Hz, matching the collection rate. Following HATO, inference runs asynchronously: each new 16-step chunk is aligned to the control tick at which its observation was taken, the first 8 steps are kept, steps whose time has passed are dropped, and the hand and wrist commands at each tick average all chunks that cover it. Each chunk is sampled with 10 DDIM steps. Predicted wrist motions are converted to flange targets with the mount transform and solved for the 7-DoF arm with Levenberg–Marquardt inverse kinematics in PyRoki.
Frequently Asked Questions
What does DexPort record during a demonstration? It records the wrist’s 6-DoF trajectory, fisheye camera observations, commanded 20-DoF joint targets, and resulting hand joint states.
How does the DexPort skill interface control dexterous hands? It uses Quest 2 controller inputs to select and modulate parameterized skills, including thumb control, power grasp, pinch grasp, two-finger grasp, and in-hand rotation.
Which tasks are used to evaluate DexPort? The evaluated tasks are vegetable pick-and-place, screwing, drawer stowing, and bimanual barcode scanning.
What limitations do the authors identify? The device’s weight limits uninterrupted use and rotation-heavy motions, and the current skills cover only part of the space of dexterous behaviors.