How can humanoids learn to interact with the diverse objects encountered in everyday life? Visual imitation learning offers a promising route: human demonstrations provide examples of the coordinated whole-body motions needed to approach, lift, carry, and drop various objects. Recent real-to-sim-to-real pipelines have enabled humanoids to acquire contextual whole-body locomotion skills directly from human videos. These advances suggest a path toward generalist humanoids that learn a broad repertoire of locomotion and manipulation skills from internet-scale video data.
However, acquiring diverse, high-quality human–object interaction videos remains a practical bottleneck to scaling visual imitation. Internet videos are abundant, but their content and framing are shaped by human viewing preferences. Filtering internet videos to find demonstrations that clearly show the person’s full body and how they interact with objects is therefore costly and impractical at scale. Yet, such interaction data exists implicitly in modern video generative models, whose learned priors over human motion and interactions can be used to synthesize diverse training videos.
The core technical challenge is to obtain physically plausible robot demonstrations despite errors introduced by both counterfactual video generation and monocular reconstruction. The key insight is to use contact signals as a shared constraint across reconstruction, retargeting, and policy learning. In the first reconstruction stage, human–object contact couples the two motions: human motion guides the object trajectory, while object contact helps correct errors in the reconstructed human pose. During retargeting, sparse contact anchors specify where robot end-effectors should contact the object. These anchors guide the refinement of noisy reconstructions into physically plausible robot–object trajectories while accommodating morphological differences. During policy learning, the anchors define contact rewards for a privileged teacher policy that co-tracks robot and object motion. The teacher is then distilled into a depth-based policy with joystick control for zero-shot sim-to-real deployment.
The contributions are: a learning paradigm that uses counterfactual video generation to scale human–object interaction experience; a contact-anchored real-to-sim and retargeting pipeline that turns monocular reconstructions into physically plausible robot–object demonstrations; and a real-to-sim-to-real pipeline that trains a unified whole-body humanoid policy to generalize across diverse objects and perform joystick-controlled pick-up, carry, and drop behaviors from onboard depth observations.
Counterfactual Interaction Video Generation
Scaling real-to-sim learning with online videos remains challenging. Although internet videos are abundant, their content and framing are shaped by human viewing preferences. Filtering for diverse clips with clear full-body views, visible human–object interactions, and stable viewpoints is therefore costly and impractical at scale. PRISM uses video-to-video generation to expand a small set of suitable real videos into diverse counterfactual interaction videos—interactions that did not occur in the source videos but could have occurred with different objects. Given a seed video and a category-level text prompt, the model is asked to replace the manipulated object while preserving the original background, lighting, camera viewpoint, and coarse task structure. This video conditioning grounds generation in a real scene and task while allowing the object and human behavior to vary.
Importantly, counterfactual variation is not limited to object geometry or appearance. When the object category, size, pose, or placement changes, its manipulation affordances also change, and human behavior adapts accordingly. The generated person may bend lower, adjust hand spacing, adapt the contact strategy to the object’s affordances, or carry the object differently. Thus, each generated clip provides an object-conditioned interaction strategy rather than merely pairing the same human motion with a new object. This allows PRISM to obtain behavior-level variations that are difficult to hard-code through geometry-level augmentation alone. Learning such adaptations from scratch would require exhaustive task-specific reinforcement-learning reward design; PRISM instead uses the video model as an offline prior over plausible human adaptations.
In practice, four real-world seed videos are recorded and used as grounded templates for video-to-video generation. SeedDance 2.0 is prompted to replace the manipulated object with a box, bin, barrel, or ball while preserving the original background, lighting, camera viewpoint, and temporal continuity. Category-level prompts allow the model to vary object geometry, appearance, and pose while adapting human behavior. With 16 samples per category, each seed yields 64 counterfactual videos, expanding four real recordings into 256 videos for the real-to-sim pipeline.
Training a Co-Tracking Teacher Policy
Humanoid object interaction is formulated as a co-tracking problem, where the policy simultaneously tracks both the humanoid motion and the dynamic object trajectory reconstructed from monocular videos. The paper refers readers to its codebase for motion-tracking training details.
Observations
Teacher observations include reference motion, robot proprioception, the previous action, and both current and target object states. Object states are represented by object pose and 3D bounding-box dimensions, making the policy explicitly aware of both the desired robot motion and the object state.
Rewards and Terminations
The reward primarily consists of robot pose tracking, object pose tracking, action rate, joint limits, and collision penalties. A contact-aware interaction reward uses the contact anchors described in the reconstruction and retargeting method. Specifically, the reward encourages the robot end-effectors to reach the desired contact positions and exceed a predefined contact-force threshold.
Reconstruction Quality from Real-to-Sim
Evaluation Metrics
For the distilled student policy, joystick commands are derived from the reference motion, and depth observations are rendered in simulation to drive the policy. A rollout is successful if the robot completes pick–carry–drop and places the object at the desired target position.
Discussion
Policies trained on OMOMO perform well across OMOMO sequences, but transfer poorly to PRISM interactions, especially with out-of-domain objects. Three main failure modes are identified: the policy struggles with unseen depth observations, overfits to close-range interactions and attempts to grasp before approaching distant objects, and fails to stabilize carry on novel object geometries.
These failures suggest that OMOMO alone lacks the variation in perception, object configuration, and interaction dynamics required for robust loco-manipulation. In contrast, PRISM-trained policies generalize across both domains, demonstrating the broader transferability of PRISM data.
Real-World Deployment
Fast-FoundationStereo substantially reduces the depth sim-to-real gap. The neck is manually tilted an unspecified angle upward from its default position, relying on camera-pose randomization during training to tolerate imprecise calibration. During bending, the neck sometimes resets to its default position or deviates from its target angle, producing out-of-distribution depth observations.
Conclusion
PRISM is a real-to-sim-to-real framework that expands a few real videos into diverse humanoid demonstrations. Contact-anchored reconstruction and retargeting turn imperfect counterfactual videos into physically plausible robot–object trajectories. The resulting unified depth-based policy picks up, carries, and drops unseen objects zero-shot in the real world. These results highlight video generation as a practical data source for generalizable whole-body interaction.
Limitations and Future Work
The pipeline delivers encouraging real-world results, yet several practical weaknesses remain.
Reconstruction and Retargeting
Monocular 4D human–object–scene recovery remains challenging, especially when preserving consistent contacts and interaction dynamics. Reconstruction errors can propagate through retargeting and compromise robot demonstrations. Although Astra was unavailable during the development of this work, such coding agents could automate and iteratively refine reconstruction from counterfactual videos. Combining these agents with contact-aware retargeting and simulation-based validation may improve the reliability of robot demonstrations.
Data Scale
Four seed videos yield 256 generated clips, 137 feasible trajectories, and 129 successful teacher rollouts. This scale is insufficient to characterize scaling behavior, with reconstruction and retargeting remaining practical bottlenecks. Future work should jointly scale generation and trajectory recovery to study how demonstration quantity and quality affect zero-shot generalization.
Object Physics
The method uses category-specific nominal masses and mesh-derived inertias, with coupled mass–inertia scaling and randomized friction. It does not identify instance-specific surface properties or internal mass distributions, which can cause sim-to-real contact mismatch.
Simulation Limitations
Dynamic objects are modeled as rigid bodies. Thus, the policy can struggle with articulated or deformable objects whose contact geometry may change during interaction. For example, foldable chairs may shift through internal joints, causing unstable grasps or loss of control.
Acknowledgments
The authors thank Chung Min Kim, Arthur Allshire, Hongsuk Choi, Isabella Yu, Junyi Zhang, Jacob Berg, Yen-Jen Wang, Sirui Chen, Charlie Cheng, Jiashun Wang, Siheng Zhao, Youjian Huang, JC Hu, Haochen Wang, Haozhi Qi, and Qitao Zhao for their support and valuable feedback.
Contact-Anchor Reliability
For two-handed carrying, 2D overlap gives contact timing; mesh intersections with the segment between palm centers give object-frame anchors. Pose and mesh errors can cause penetration and inverse-kinematics failure. The method retains non-penetrating inverse-kinematics solutions and full-trajectory teacher rollouts, without formal contact or feasibility guarantees.
Pipeline Runtime
The paper reports mean times per 8-second clip in a referenced table, but the table’s data values are not included in the provided text. Reconstruction and retargeting use one NVIDIA L40S GPU; video-to-video generation latency is recorded separately.
Additional Interaction Examples
The paper presents ball-rolling and door-opening reconstructions given suitable contact points and object representations. The door uses SAM3D for geometry and Codex for articulation. These qualitative examples are not policy or real-world evaluations; tool use, regrasping, broader deformable-object manipulation, and extreme dynamics such as those in ExoRecon remain untested.
Frequently Asked Questions
How does PRISM generate diverse interaction demonstrations? It uses video-to-video generation to replace objects in real seed videos while preserving the scene and adapting human behavior to the object.
What role do contact anchors play in the pipeline? They guide contact-aware reconstruction and retargeting, and define contact rewards for the teacher policy.
How many counterfactual videos were generated? Four seed videos were expanded into 256 generated videos using 16 samples per object category.
What are key limitations of the approach? Monocular reconstruction errors, limited data scale, uncertain object physics, and rigid-body simulation can all limit performance.