Researchers from NVIDIA and the University of Washington built GRAIL, a fully digital pipeline that generates over 20,000 physically plausible humanoid loco-manipulation sequences from 3D assets and video foundation models. This approach eliminates the need for costly teleoperation or motion capture, and the generated data directly transfers to real robots, achieving an 84% pick-up success rate and 90% stair-climbing success on a Unitree G1.
What the Researchers Built
GRAIL is a data-generation framework that turns a single video clip of a human performing an interaction — like picking up a box or climbing stairs — into a complete, robot-ready training sequence. Unlike prior approaches that try to reconstruct ambiguous 4D scenes from uncontrolled video, GRAIL first fully specifies the 3D asset (object and scene geometry), camera parameters, metric scale, and a robot-proportioned character. It then uses video foundation models (VFMs) as priors to recover the precise interaction within this known metric frame.
The pipeline produces over 20,000 physically plausible sequences covering tasks like object pick-up, stair climbing, slope traversal, and sitting. Each sequence is automatically checked for physical plausibility — collisions, ground penetration, and balance — and only valid sequences move forward. The output includes robot-trackable trajectories that are already aligned with simulation, enabling direct downstream sim-to-real policy training. The team validated the pipeline end-to-end by deploying egocentric visual policies on a real Unitree G1 humanoid robot.
Key Results
GRAIL demonstrates that its generated data produces real-world capable policies. The key benchmark numbers come from physical deployment on a Unitree G1:
| Task | Real-World Success Rate |
|---|---|
| Object pick-up | 84% |
| Stair climbing | 90% |
Beyond these deployment results, the paper evaluates the quality of the generated 4D HOI sequences. GRAIL’s sequences are more physically executable than existing generation baselines — the team found that fewer sequences needed to be discarded due to collisions or unrealistic kinematics. The task-general tracking policies also show strong amortization: once trained on a pool of related 4D HOI sequences, the same tracker can handle new motions within that family without per-sequence retraining. The entire data generation pipeline runs fully digitally, requiring no physical data collection beyond a single video of a human performing the task.
How It Works
GRAIL operates in three stages. First, data generation: given a 3D asset (object + scene) and a video foundation model (VFM) that suggests an interaction, the pipeline reconstructs a 4D human-object interaction (HOI) sequence within a fully specified metric world. The scene geometry, camera, scale, and a robot-proportioned character are all defined before the VFM is queried — this avoids the ambiguity of reconstructing from uncontrolled video. The output is a complete trajectory of the humanoid’s joints, object pose, and contact states.
Second, tracker training: GRAIL pairs the reconstructed data with complementary task-general trackers. For manipulation tasks (e.g., pick-up), an object-aware latent adaptor fine-tunes the low-level controller using the object’s geometric features. For scene-level interactions (e.g., stairs, slopes, sitting), the controller is fine-tuned end-to-end with a height-map encoder and an auxiliary kinematic decoder. Both trackers output latent tokens that drive the robot’s whole-body motion without per-sequence manual tuning.
Third, sim-to-real deployment: The object-aware and scene-aware trackers are distilled into separate egocentric visual policies using head-camera RGB inputs (from a Luxonis OAK-D camera). These policies output latent tokens for the SONIC controller. Domain randomization during training ensures transfer to the real world. On the real Unitree G1, a desktop with an NVIDIA RTX 5090 GPU runs inference at 10 Hz, streaming visual and proprioceptive data over a network connection. This pipeline allowed the robot to autonomously pick up objects and climb stairs in the real world with high success.
Why This Matters for Robotics
GRAIL tackles the data bottleneck that has limited humanoid robot capabilities. Current methods for generating loco-manipulation data rely on expensive teleoperation setups, motion capture facilities, or painstaking hand-coded trajectories — none of which scale economically. GRAIL’s fully digital pipeline cuts that cost dramatically: a single video of a human performing a task is enough to generate thousands of robot-ready training sequences.
For companies looking to deploy humanoid robots in warehouses, factories, or service environments, this means faster skill acquisition at lower cost. Instead of hiring operators to teleoperate a robot for weeks, operators can simply record reference videos and let GRAIL generate the training data. The fact that the same pipeline works for both manipulation (picking objects) and locomotion (climbing stairs) makes it particularly valuable for general-purpose humanoids. For buyers, this technology directly impacts the ROI of platforms like the Unitree G1. Explore available humanoid robots on Robot Overflow or compare with warehouse robots for task-specific needs.
Limitations and Open Questions
GRAIL inherits several constraints. The pipeline requires a high-quality 3D asset of the object and scene — it cannot work from video alone. The video foundation model must also correctly follow the requested interaction, and reconstruction quality degrades under severe occlusion, fast motion, or inconsistent object appearance. The failure-filtering step discards a non-trivial fraction of generated sequences, reducing overall yield. Additionally, the task-general tracking policies still need retraining or fine-tuning when the motion family changes substantially — for example, switching from walking on flat ground to crawling under obstacles. Real-world validation was limited to two tasks on one robot platform; scaling to more diverse motions and multiple robot hardware remains open.
Frequently Asked Questions
What does GRAIL stand for? GRAIL stands for Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors — a fully digital pipeline that produces robot training data from a single reference video and a 3D model of the object and scene.
How does GRAIL generate training data without teleoperation? It uses a video foundation model as an interaction prior, then reconstructs the human-robot interaction in a fully specified 3D scene with known geometry, scale, and camera parameters — eliminating the need for physical data collection or hand-coded trajectories.
What real-world success rates did the Unitree G1 achieve with GRAIL data? The robot achieved 84% success on object pick-up tasks and 90% success on stair climbing in the real world, running egocentric visual policies trained on GRAIL-generated sequences with domain randomization.
What are the main limitations of GRAIL? It requires 3D assets for every object and scene, the video foundation model must accurately suggest the interaction, and reconstruction quality drops under occlusion or fast motion. The generated trackers also need retraining for substantially different motion families.
Conclusion
GRAIL offers a scalable, digital-only solution for generating humanoid loco-manipulation training data, validated by real-world deployment on a Unitree G1. By combining video foundation model priors with precisely specified 3D assets, it reduces the cost and complexity of teaching robots new physical skills. This work points toward a future where humanoid robots can learn complex tasks from a single video demonstration.