Two eye-in-hand cameras mounted on separate robot arms now share a calibrated 3D workspace, producing synchronized RGB, depth, infrared, and robot-motion data around tabletop objects. The DARP dataset gives researchers and robotics engineers a way to test multi-view perception using real kinematics and complementary viewpoints, rather than relying only on fixed cameras, synthetic scenes, or guessed camera poses.
What did the researchers build?
The researchers built DARP, a dual-arm dataset for studying how robots combine multiple camera views of the same object. The platform uses two fixed-base Unitree Z1 Pro manipulators, each with six degrees of freedom and an eye-in-hand RGB-D-IR camera. One arm is positioned on the left side of a shared tabletop workspace and the other on the right, with the robot bases approximately 1.3986 metres apart.
Each camera moves around an object from its own side while recording colour images, depth, infrared data, camera settings, robot joint states, and calibration information. Because the cameras are attached to moving wrists, every observation has a corresponding robot pose. This lets each depth frame be transformed into a common metric world coordinate system.
The collection process does not require a turntable, marked placement area, or manually measured object location. An operator places one tabletop object somewhere inside the shared workspace, after which the system searches for it, confirms its location from both arms, plans viewpoints, records the synchronized data, and checks whether the survey is valid.
The current release contains ten unique tabletop objects. Its main contribution is not simply a larger image collection; it is the connection between sensor data, robot motion, and calibration in a single dual-arm acquisition setup.
What were the key results?
DARP’s geometric evaluation shows that the two independently moving cameras can be aligned without visually re-registering every frame. Each arm sees different portions of an object, while overlapping regions remain reasonably consistent after robot poses and calibration transforms place both streams into the same world frame.
The researchers tested the fused surface using RGB-D frames that were deliberately excluded from reconstruction. Across 224 held-out keyframes, the evaluation covered 1,563,466 three-dimensional query points. The median point-to-mesh distance was 2.13 millimetres, and the root-mean-square error was 4.04 millimetres.
| Held-out evaluation metric | Result |
|---|---|
| RGB-D keyframes | 224 |
| 3D query points | 1,563,466 |
| Median point-to-mesh distance | 2.13 mm |
| Root-mean-square error | 4.04 mm |
| 90th-percentile distance | 6.38 mm |
| Points within 5 mm | 83.82% |
| Points within 10 mm | 96.56% |
These figures measure agreement with recorded observations, not absolute accuracy against a precision scanner. Even so, the results demonstrate that robot-derived camera poses are consistent enough to support metric fusion and held-out surface checks.
Qualitative reconstructions also show the practical benefit of the second arm. A single arm leaves opposite-facing and self-occluded regions missing, while the second viewpoint adds complementary surface measurements.
How does the calibrated dual-arm system work?
The system relies on a calibration chain that connects each camera to its robot wrist and connects both robot bases to a shared world frame. ChArUco calibration boards provide coded marker detections and chessboard corners for estimating camera intrinsics and rigid transforms. Hand-eye calibration then estimates the fixed relationship between each camera and its wrist.
During recording, synchronized joint states are combined with forward kinematics. Forward kinematics converts the measured angles of a robot’s joints into the position and orientation of its wrist. Applying the wrist-to-camera transform produces a camera pose for every observation. The system therefore obtains sensor poses primarily from robot calibration and motion data rather than tracking visual features from frame to frame.
Object acquisition follows several stages:
- Object search: An eye-in-hand camera performs a structured pan across the tabletop until depth data supports an object estimate.
- Cross-arm confirmation: Both arms independently check the estimated object centre and workspace validity.
- Envelope creation: The system combines valid centre estimates and retains a conservative object radius to reject table and background points.
- Adaptive viewpoint planning: Each arm moves through changing viewpoints on its accessible side, producing complementary rather than identical observations.
- Synchronized recording: RGB, depth, infrared, camera metadata, joint states, and calibration-related robot information are retained together.
- World-frame fusion: Metric depth is back-projected into 3D, transformed into the shared frame, filtered, and accumulated across both arms.
- Surface generation: Statistical outlier removal and local depth checks clean the cloud before multi-scale ball pivoting creates a measured surface mesh.
Ball pivoting generates triangles only where the point cloud provides supporting measurements. The pipeline does not fill unseen undersides or hallucinate hidden geometry. This distinction makes DARP useful for studying real partial-view reconstruction, where missing data is part of the problem rather than an error concealed by synthetic completion.
Why does this matter for robotics?
Many robot vision datasets provide images and depth, but fewer preserve the complete relationship between moving sensors and robot motion. DARP gives perception engineers a controlled way to test whether a system can combine observations from independently moving manipulators while retaining the raw multimodal evidence needed for alternative algorithms.
That matters for bin picking, collaborative manipulation, inspection, object modelling, and robotic grasp planning. A robot approaching an object from one side often sees only a partial surface. A second arm or mobile camera can contribute a different view, but only if the measurements are aligned accurately enough for geometry, segmentation, or grasp planning.
The dataset also supports research beyond geometric fusion. Its RGB-D-IR streams can support object recognition, depth completion, sensor-fusion models, viewpoint selection, and feature-level collaborative perception. Since robot states are preserved, future systems can learn to connect camera observations with reachability, arm configuration, and viewpoint quality.
For operations teams evaluating automation, the work highlights why camera placement and calibration are as important as model selection. A facility considering used cobots for sale or used industrial robots needs repeatable methods for measuring how robot-mounted sensors behave in the actual workspace.
What are the limitations and open questions?
The dataset contains ten unique tabletop objects from a single laboratory setup, so it does not represent the full variation of industrial parts, clutter, materials, lighting, or workspace layouts. The reconstructed meshes contain only surfaces observed by at least one camera; undersides and strongly occluded regions remain incomplete.
Performance also depends on calibration stability. Moving a camera bracket, changing a robot-base position, altering sensor mounting, or introducing joint-zero errors can reduce alignment quality and require recalibration. The point-to-mesh results measure repeatability against held-out RGB-D observations, not accuracy against an independent high-precision model.
Open questions include how well the approach transfers to reflective or transparent objects, cluttered scenes, larger workspaces, faster arm motion, and production environments. Future learning systems could also use DARP for shape completion, but those predictions would need to remain clearly separate from the dataset’s measured geometry.
Frequently Asked Questions
What is DARP?
DARP is a calibrated dual-arm RGB-D-IR dataset for multi-view robotic perception. It links complementary camera observations with robot poses, joint states, and calibration data.
What robots and sensors were used?
The platform uses two six-degree-of-freedom Unitree Z1 Pro arms with wrist-mounted RGB-D-IR cameras. The fixed bases face one another across a shared tabletop workspace.
How accurate is the fused reconstruction?
On 224 held-out keyframes, the median point-to-mesh distance was 2.13 mm and the RMSE was 4.04 mm. A total of 96.56% of evaluated points fell within 10 mm of the measured surface.
Does DARP reconstruct hidden object surfaces?
No. Its geometric pipeline preserves missing or occluded regions instead of inventing unseen geometry through shape completion.
Conclusion
DARP connects dual-arm robot motion, calibrated eye-in-hand cameras, and synchronized RGB-D-IR observations in a shared metric frame. Its held-out evaluation shows that complementary views can be fused with millimetre-scale observation agreement, creating a practical foundation for multi-view robotic perception research.