Ego4D and Ego-Exo4D are among the largest efforts in egocentric data collection, providing thousands of hours of video across a wide range of activities and participants. Datasets of this scale open the door to pipelines for skill learning and assessment, procedural activity understanding, embodied AI and robot imitation learning, and coaching or tutoring systems that give feedback on physical performance.
Introduction
Ego4D and Ego-Exo4D are among the largest efforts in egocentric data collection, providing thousands of hours of video across a wide range of activities and participants. Datasets of this scale open the door to pipelines for skill learning and assessment, procedural activity understanding, embodied AI and robot imitation learning, and coaching or tutoring systems that give feedback on physical performance.
Ego-Exo4D in particular is a rich dataset, with extensive capture including an egocentric sensor alongside multiple exocentric cameras, all synchronized and calibrated. This multi-view setup should, in principle, significantly improve the quality of 3D human perception, since the exocentric cameras observe the full body of the person performing the activity. However, the dataset’s existing 3D annotations are sparse, and current 3D human reconstruction systems can be brittle at this scale.
To enable broader use of the dataset, we develop a pipeline for recovering the 4D human motion of Ego-Exo4D’s captures. Our approach builds on state-of-the-art methods for 3D human pose reconstruction, adapted to take advantage of Ego-Exo4D’s synchronized multi-camera setup. We release the resulting SMPL-H motion sequences as the Ego-Exo4D-HM dataset, along with our processing pipeline, so future work can use this data directly rather than reprocessing the raw captures.
Background
Previous work has independently considered the problem of egocentric body pose recovery. EgoEgo relies only on the SLAM trajectory of the head-mounted camera, without access to image observations, while more recent work, EgoAllo, incorporates hand observations estimated from egocentric images. The exocentric cameras of Ego-Exo4D simplify the problem in this setting, since they are observing the body from multiple views.
Beyond embodied AI, understanding and modeling human activity is also central to skill assessment, coaching, and video forecasting. Previous work has used estimates of 3D motion in egocentric settings to provide actionable feedback, predict future interactions, forecast hand motion, and edit novice motion toward an expert’s skill level. We anticipate that populating ego-exo captures with dense 3D motion estimates at scale will be valuable for supporting these methods.
Methods
Given synchronized egocentric and exocentric video from the Ego-Exo4D activity dataset, our goal is to reconstruct world-frame 3D body and hand motion of the person performing the activity. We build off SLAHMR, which can be naturally extended to multi-view settings. Following SLAHMR, we represent the person’s state at each timestep using a parametric human-body representation.
The person’s state at each timestep is represented by the parameters needed to describe the body and hand pose, shape, and world-frame motion.
Our mesh recovery pipeline modifies SLAHMR to leverage Ego-Exo4D’s exocentric calibrations and egocentric SLAM estimates. Concretely, our pipeline consists of three stages:
- Single-view pose estimation
- Triangulation
- SMPL-H optimization
Single-View Pose Estimation
Takes frequently contain bystanders, so we first identify the camera wearer in each exocentric view. We run a Mask R-CNN detector with a RegNetY-4GF backbone and keep the detection whose box contains the wearer’s projected 3D head position, given by the Aria SLAM trajectory.
For each selected box, we run ViTPose to obtain human body keypoints and use HaMeR to obtain hand keypoints. In total, we obtain 67 hand and body keypoints.
Triangulation
Each of the 67 keypoints is triangulated independently per frame by minimizing reprojection error over all calibrated views, using nonlinear least squares over the 3D point. At least two views are required.
For the hand and wrist keypoints, which are more prone to occlusion and misdetection, we additionally run RANSAC over views to reject outlier detections.
Curated Dataset
We ran our pipeline on a subset of approximately 3,200 videos from the Ego-Exo4D dataset. From here, we apply a quality filter based on two per-take criteria.
Reprojection Self-Consistency
We reproject the final optimized 3D joints through the calibrated cameras and measure the pixel distance to the corresponding 2D detections. A take is flagged if more than 10% of these samples exceed 50 pixels.
Triangulation Coverage
A take is flagged if fewer than 50% of its keypoint observations were successfully triangulated. This catches takes reconstructed from too few agreeing views.
Quantitative Results
We evaluate reconstruction accuracy against the ground-truth 3D body and hand keypoint annotations provided by Ego-Exo4D. Since our reconstructions live directly in the dataset’s metric world frame, we report global MPJPE, the mean Euclidean distance between predicted and annotated 3D joints without any alignment.
Our method achieves:
- Global body MPJPE: 56.21 mm over 845 annotated takes
- Global hand MPJPE: 51.59 mm over 190 annotated takes
Comparison with MAMMA
We additionally compare qualitatively with MAMMA, a recent multi-view mesh recovery method. On the partial body views, occlusions, and truncations common in Ego-Exo4D’s captures, we find that our pipeline produces more robust reconstructions than MAMMA.
Acknowledgements
This project received computing support on the Lonestar6 GPU Cluster through the Center for Generative AI and the Texas Advanced Computing Center at the University of Texas at Austin.
Frequently Asked Questions
What is the Ego-Exo4D-HM dataset? It is a dataset of SMPL-H motion sequences recovered from Ego-Exo4D captures, released with the processing pipeline.
What data does the reconstruction pipeline use? It uses synchronized egocentric and exocentric video, exocentric camera calibrations, and egocentric SLAM estimates.
How many keypoints are reconstructed? The pipeline obtains and triangulates 67 hand and body keypoints.
What global MPJPE does the method achieve? It achieves 56.21 mm for body keypoints and 51.59 mm for hand keypoints.