DexVerse is a modular benchmark designed for multi-task, multi-embodiment dexterous manipulation. It unifies diverse task categories, multiple arm-hand embodiments, configurable visual variation, VR-based teleoperation, multi-modal demonstrations, and representative policy evaluation. The benchmark provides a challenging and extensible testbed for studying contact-rich dexterous control, visuomotor robustness, and embodiment-aware robot learning.
Modular Environment Design
DexVerse uses a configuration-driven design to specify and instantiate manipulation environments. Each environment is defined by a set of structured components, including the scene layout, object assets, robot embodiment, observation and action interfaces, initialization rules, success conditions, and randomization settings. Tasks within the same family share reusable templates for common logic such as asset loading, state initialization, reset events, and success checking, while task-specific parameters define the manipulated objects, target states, sampling ranges, and completion thresholds. This design reduces duplicated implementation across related environments and makes task variants easier to construct and maintain.
DexVerse builds on the manager-based environment interface of Isaac Lab, where observations, actions, events, terminations, and optional reward terms are specified through configuration classes and executed by a shared simulation loop. Most environment parameters can be adjusted through configuration overrides, enabling controlled changes to initialization ranges, camera settings, randomization options, or success thresholds without modifying the core environment code. For robot embodiments, DexVerse additionally provides a compact specification for selecting arm-hand combinations, simplifying the instantiation of feasible embodiment variants.
Robot Embodiments
DexVerse specifies each robot embodiment through a robot configuration, which defines the arm and hand models, initial pose, action interface, controller parameters, and embodiment-specific constants. As long as a robot implements the required interface, it can be instantiated in a task without rewriting the task-specific environment logic. For physically feasible task-embodiment combinations, users can override the default robot choice, such as replacing a single-arm setup with a bimanual setup or switching to a different arm-hand pair.
The current benchmark supports 3 robot arms (Franka Research 3, UR10e, and xArm 7) and 6 dexterous hands (Sharpa Wave, WUJI Hand, Shadow Hand, Inspire Hand, Allegro Hand, and LEAP Hand), covering diverse kinematics, degrees of freedom, joint limits, actuation ranges, and hand morphologies. Each hand also has a floating variant, where the wrist is directly controlled by prismatic and revolute joints.
Visual Variation
DexVerse provides configurable visual variation as part of its environment specification. Each environment uses a fixed default appearance when visual randomization is disabled. When enabled, visual properties are sampled at reset from predefined libraries, including object materials, table materials, lighting conditions, background skyboxes, exposure, and color-temperature settings. DexVerse also supports camera-viewpoint changes, allowing the same task to appear under different observation conditions while preserving the task objective and success conditions.
Beyond visual variation, DexVerse also supports non-visual variations, including object initial poses, task sampling ranges, proprioceptive and object-state perturbations, and dynamics parameters. Visual and non-visual variations can be enabled independently or jointly, making appearance changes explicit while preserving standard environment variability for manipulation tasks.
Dataset and Evaluation
The pipeline is designed to reduce embodiment-specific changes when adapting to new arm-hand platforms. Adding new robot arms typically requires updating the end-effector frame, initial poses, and low-level controller parameters, while adding dexterous hands requires configuring the hand URDF with necessary keypoints, correspondence links, and retargeting scales.
Dataset Statistics. DexVerse provides teleoperation demonstrations for most of the task suites. For each of the 56 single-goal tasks, we collect 55 demonstrations: 50 with the Shadow Hand and one with each of the other five hand embodiments. For each of the 5 long-horizon tasks, we collect 20 demonstrations. This results in a total of 3,180 demonstration trajectories.
Each demonstration is stored as a sequence of action-state pairs recorded during teleoperation. We provide the replay utility that restores the recorded simulator states and queries the environment locally to regenerate the requested observation terms. This design is important because physics simulation can diverge across machines due to differences in physics computation, hardware, and floating-point rounding. Direct state replay avoids accumulated rollout drift and makes demonstrations more portable across local setups.
The action-state format also keeps the dataset compact. The replay mechanism also makes modifying observation presets, adding observation terms, or changing camera-based inputs more flexible without requiring a separate copy of the demonstration dataset.
Multi-Goal Composite Tasks
An environment is marked successful when both of the following hold at the same time: - The addon object is lifted at least a specified threshold above its reset height, regardless of which addon task it originally comes from. - The primary task's articulated joint also reaches its own single-task success threshold a task-specific threshold.
For when the primary object is the mug, the same lift-then-pour procedure applies. Both conditions must hold simultaneously for the task to count as a success. The lift threshold is a single fixed value shared across all 39 variants, while the task-specific threshold matches each primary task's own standalone threshold.
Observation-Mode Presets
While the user is free to select the observation groups to use, DexVerse also defines some observation-mode presets, each of which enables a fixed set of groups and disables every other managed group. A preset is applied to an environment's ObservationsCfg through a config field. The privileged group is never enabled by any preset since it contains information that is hardly observable in the real world and should only be intentionally enabled by the user.
Conclusion
We presented DexVerse, a modular benchmark for multi-task, multi-embodiment dexterous manipulation that unifies diverse task categories, multiple arm-hand embodiments, configurable visual variation, VR-based teleoperation, multi-modal demonstrations, and representative policy evaluation. Our experiments show that current imitation-learning and vision-language-action policies remain far from solving general dexterous manipulation, particularly for precise contact, bimanual coordination, functional tool use, and robust interaction with complex objects. These findings establish DexVerse as a challenging and extensible testbed for studying contact-rich dexterous control, visuomotor robustness, and embodiment-aware robot learning.
Limitation and Future Work. The current release focuses on building a broad, reproducible, and multi-task, multi-embodiment benchmark. Future extensions will study real-robot transfer, expand demonstrations across more embodiments and task families, and provide broader standardization for cross-task and cross-embodiment evaluation. These directions will further strengthen DexVerse as a foundation for developing general, robust, and transferable dexterous manipulation policies.
Frequently Asked Questions
What robot arms and hands does DexVerse support? DexVerse supports 3 robot arms (Franka Research 3, UR10e, xArm 7) and 6 dexterous hands (Sharpa Wave, WUJI Hand, Shadow Hand, Inspire Hand, Allegro Hand, LEAP Hand), with each hand also available in a floating variant.
How many demonstration trajectories does the benchmark include? The dataset contains 3,180 demonstration trajectories total — 55 per single-goal task (50 with Shadow Hand, one per other hand) and 20 per long-horizon task.
How are demonstrations stored to avoid simulation drift? Each demonstration is stored as a sequence of action-state pairs, and a replay utility restores recorded simulator states to regenerate observation terms locally, avoiding accumulated rollout drift across different machines.
What visual randomization options are available? DexVerse supports configurable object materials, table materials, lighting conditions, background skyboxes, exposure, color-temperature settings, and camera-viewpoint changes, all adjustable independently or jointly with non-visual variations.