Three synchronized stereo endoscopes gave surgical robots a clearer 3D view of tissue, but the best reconstruction method depended on how many viewpoints were available. The MV-dVRK benchmark pairs real robotic surgery scenes with validated surface geometry and camera poses, showing that stereo leads with two endoscopes while geometry-refined multi-view systems win after a third viewpoint is added.
What Did the Researchers Build?
MV-dVRK adds a second stereo endoscope to the da Vinci Research Kit, or dVRK, using an independent robotic arm. The benchmark setup then adds a third stereo endoscope on a fixed mount. All three viewpoints capture synchronized image pairs, giving researchers up to six images of the same surgical scene at each instant.
The team collected data during mock surgery on freshly excised abdominal organs from two adult pigs. Two experienced visceral surgeons performed representative tasks, including gallbladder removal, tubular tissue dissection, and small-bowel reconnection. The recordings include both static scenes, used to build detailed geometric references, and dynamic sequences involving camera movement, tools, and tissue deformation.
The static portion contains eight scenes and 80 synchronized test samples. Each test sample includes three stereo pairs selected to vary camera positions, overlap between views, lighting, robot configurations, and tool placement. For each static scene, 105 carefully selected images were used to create a dense reference reconstruction.
That reference combines the tissue surface and the pose of every camera in one coordinate system. An industrial 3D scanner independently checked the reconstructed geometry, providing a practical validation step rather than relying only on a computer vision estimate.
What Were the Key Results?
The benchmark exposes a clear trade-off between stereo and multi-view reconstruction. With only two endoscopes, calibrated stereo produced more accurate geometry in the shared reference view than every tested multi-view method. Stereo also covered substantially more tissue than monocular reconstruction, even when the monocular system processed a long sequence from a moving camera.
Adding a third viewpoint changed the result. Geometry-refined multi-view methods, which jointly optimize camera poses and 3D structure, reduced the average reconstruction error in the common view by 1.9 millimeters, a 54% relative improvement. With all six synchronized images available, these methods achieved 66–67% coverage within 1 millimeter, compared with 28–44% for feed-forward models.
| Comparison | Benchmark result | Practical meaning |
|---|---|---|
| Stereo versus two-frame monocular reconstruction | 27 percentage-point coverage advantage; 95% confidence interval of 20–34 points across all 8 scenes | A stereo pair captures more usable surface immediately |
| Stereo versus 50 monocular frames | 15 percentage-point coverage advantage; confidence interval of 5.8–25 points in 6 of 8 scenes | Camera motion over time does not fully replace simultaneous stereo |
| Third viewpoint for geometry-refined multi-view methods | 1.9 mm lower mean error and 54% relative improvement | Additional cross-view support improves shared-view accuracy |
| Six synchronized views versus feed-forward models | 66–67% versus 28–44% coverage within 1 mm | Explicit geometric optimization benefits from enough viewpoints |
The results do not mean that more cameras always improve every metric. The second endoscope’s view showed a more variable improvement because tissue overlap and camera geometry differed between scenes.
How Does the Benchmark Work?
The benchmark tests several reconstruction strategies under the same synchronized input conditions. A monocular method, such as Endo3R, estimates 3D structure from one endoscope over either two or 50 frames. A stereo method, such as FoundationStereo, uses the calibrated left and right cameras in one endoscope to infer depth from corresponding image features.
The multi-stereo setup runs stereo reconstruction independently for each endoscope. The resulting point clouds are then aligned with visual correspondences and rigid or scale-flexible transformations. This approach combines several stereo reconstructions but does not jointly optimize all images and all scene points at once.
Multi-view methods instead process images from multiple endoscopes in a shared geometric framework. Geometry-refined systems use the initial reconstruction and then optimize camera poses and scene structure together. That optimization becomes more useful when a third endoscope supplies additional cross-view matches and reduces ambiguities caused by large camera baselines or limited overlap.
Before scoring, every prediction is registered to the reference reconstruction. The registration uses a robust similarity transformation, known as Sim(3), which can correct rotation, translation, and global scale. This design evaluates the shape and relative camera geometry without penalizing a method solely because it estimates a different absolute scale.
Three metrics capture different aspects of performance:
- Ground-truth coverage: the fraction of reference surface points with a predicted point within 1 millimeter. A 5-millimeter version is also reported.
- Mean 3D error: the distance between predicted points and reference points corresponding to the same pixels in the shared E1L camera view.
- Relative pose error: the accuracy of the estimated transforms between cameras, measured in millimeters for translation and degrees for rotation.
The E1L camera provides the common comparison area because it has the most complete reference surface and is visible across the tested input configurations.
Why Does This Matter for Robotics?
Robotic surgery operates in a difficult perception regime. Endoscopes must fit through narrow access points, instruments and anatomy frequently block one another, and only a small number of cameras can be placed inside the body. The result is a sparse-view problem: cameras are separated by large baselines, while their images can have limited overlap.
MV-dVRK shows that camera count and algorithm choice should be designed together. Two endoscopes favor a calibrated stereo pipeline because it extracts reliable depth from a tightly coupled pair. A third viewpoint creates enough shared evidence for joint multi-view optimization to overtake independently registered stereo.
That finding could guide future surgical robots toward modular sensing systems rather than a single universal reconstruction algorithm. A platform could use fast stereo for low-latency control, then invoke geometry refinement when extra synchronized views are available and computation permits.
The lesson also applies beyond medicine. Teams evaluating the broader market of used industrial robots can treat coordinated cameras, calibration, and robot motion as one perception system rather than separate hardware choices. In any multi-arm workspace, better coverage depends on where sensors are placed and how their observations are fused.
What Are the Limitations and Open Questions?
The strongest ground-truth results come from static tissue, because dense reference geometry is difficult to obtain while tissue deforms. The dynamic MV-dVRK sequences include camera motion and changing anatomy but do not provide equivalent dense surface ground truth.
The static benchmark covers eight scenes from two porcine specimens and three surgical tasks. That supports controlled comparisons, but performance differences could change with other anatomies, procedures, lighting conditions, or in-vivo environments. The reference surface is reconstructed rather than directly measured, although an industrial scanner independently validates its geometry.
Camera poses also come from dense structure-from-motion reconstruction and cannot be fully checked externally without introducing additional calibration biases. Practical deployment still requires faster dense matching, lower-latency optimization, and methods that account for known stereo-rig geometry, robot kinematics, and tissue deformation.
Frequently Asked Questions
What is MV-dVRK?
MV-dVRK is a robotic surgery dataset and benchmark built around three synchronized stereo endoscopes. It evaluates sparse-view 3D reconstruction using real ex-vivo surgical scenes.
Why are three endoscopes better than two for some methods?
A third viewpoint provides additional cross-view matches and helps jointly optimized methods refine camera poses and tissue geometry. With only two endoscopes, calibrated stereo remained more accurate in the shared reference view.
Does MV-dVRK measure performance on moving tissue?
The collection includes dynamic sequences with camera motion and tissue deformation, but dense reference geometry is available only for static scenes. Dynamic data therefore supports future perception research rather than the same detailed geometric scoring.
What should a surgical robot use today: stereo or multi-view reconstruction?
The benchmark favors calibrated stereo when only two endoscopes are available and geometry-refined multi-view reconstruction when three synchronized viewpoints provide sufficient overlap. Latency, camera placement, calibration quality, and the amount of visible tissue remain important system-level factors.
Conclusion
MV-dVRK establishes a realistic testbed for measuring how surgical 3D perception changes as synchronized viewpoints are added. Its results show a practical crossover: stereo is strongest in the sparsest setup, while jointly optimized multi-view reconstruction becomes more accurate and complete with a third endoscope.