Humans can grasp hundreds of novel objects in diverse and complex scenes, yet existing robot grasping systems are limited to a small set of prescribed behaviors on tabletop bins or sparse clutter. Scaling up robot grasping requires a model that understands the full diversity of human grasping across objects, environments, and tasks. We present HUG, a flow-matching model trained on 1M human grasps from egocentric stereo videos, that generates diverse, human-like grasps for any object in any scene from a single RGB-D image.
Dataset and Collection
To train HUG, we built 1M-HUGs, a dataset of 1 million real-world human grasps captured across over 400 hours of egocentric video from 70 participants in diverse indoor environments. Grasps are labeled automatically using a vision-language model to identify and segment grasped objects, followed by automated frame selection and quality control through a web annotation interface.
Method
HUG is a flow-matching model that, trained on real-world human grasps, generates diverse natural grasps for any user-specified object in a single RGB-D image from a stereo camera. The model processes a point cloud of the scene and outputs a MANO hand pose that represents the predicted grasp.
Training
During training, the weight concentrates the geometric loss on near-clean steps where the predicted grasp is most meaningful. This approach ensures the model learns to produce precise, physically plausible grasps.
Experiments
We introduce HUG-Bench for standardized evaluation, then evaluate HUG in simulation and the real world.
Real-World Evaluation Design
Objects are placed at a single location for tabletop trials, varying only position and rotation slightly across trials. Clutter is avoided in tabletop experiments to ensure repeatability and fair comparison with baseline methods. In-the-wild experiments in diverse cluttered settings demonstrate robustness to real-world clutter.
Retargeting to Robot Hands
HUG predicts a MANO grasp, which is mapped to target robot hands at deployment. We retarget to the Ability hand with AnyTeleop and to the WUJI hand with WUJI retargeting.
Qualitative Observations
The easiest objects are rounded and convex, sized to fit the hand (e.g., pear, pineapple, hacky sack), which afford many stable enveloping grasps. The hardest objects are too large for the hand to wrap (e.g., football, wipe dispenser) or irregular and articulated objects difficult to grasp open-loop (e.g., nail clipper, headphones).
HUG is robust to changes in lighting, object rotation, and viewpoint, operating on both color and grayscale stereo cameras. Successful grasps are also observed on reflective surfaces where stereo depth is noisier.
Comparison against a strong gripper policy shows that the benefit of dexterity is object-dependent. On large or irregular objects, an antipodal gripper finds no stable pair of opposing faces and tends to slip, whereas an enveloping multi-finger grasp cages the object. On heavy objects whose graspable feature is offset from the center of mass, a multi-finger grasp distributes contacts and resists gravitational torque. On small, thin, or pinchable objects the gripper remains competitive.
Automated Labeling
The first three stages run without human input: identifying the grasped object, segmenting it across the recording, and selecting the grasp frame. The grasped object is localized with a vision-language model that compares a "before" frame against four "after" frames and returns points covering all parts of the object. A second VLM call verifies each point lies on the grasped object. The cost is roughly $0.003 per recording.
Quality Control
Every recording is reviewed in a web annotation app where a reviewer can re-segment the mask, re-select the grasp frame, and either approve the recording or mark it as having no stable grasp.
Point Cloud Crop Radius
Before encoding, the back-projected point cloud is cropped to a ball of fixed radius around the 3D query point.
Camera Generalization
The camera intrinsics enter HUG only through geometric operations: back-projecting a pixel with depth to a metric point.
Limitations
While HUG achieves state-of-the-art performance, it inherits limitations from the training data. The model may struggle with extremely rare objects, very small or large objects, or grasps requiring fine finger dexterity that are underrepresented in the dataset. Future work could explore incorporating tactile feedback and fine-tuning on robot data.
Frequently Asked Questions
What is the key innovation of HUG? HUG is a flow-matching model trained on 1 million real-world human grasps from egocentric video, enabling diverse, human-like grasps for any object in any scene from a single RGB-D image.
How was the training data collected? The 1M-HUGs dataset was collected from over 400 hours of egocentric stereo video from 70 participants in diverse indoor environments, with automated labeling using a vision-language model and quality control through a web annotation interface.
How does HUG compare to gripper-based grasping? On large, irregular, or heavy objects with off-center mass, multi-finger grasping significantly outperforms grippers by distributing contacts and resisting torque, while grippers remain competitive on small, thin, or pinchable objects.
What are the computational requirements for HUG? The automated labeling pipeline costs roughly $0.003 per recording using a vision-language model, and the trained model runs efficiently on standard GPU hardware for real-time grasp generation.