Connecting egocentric observations to whole-body action is a fundamental challenge for humanoid robots operating in human-centered environments. A humanoid must perceive the world from its own viewpoint, identify task-relevant objects, navigate toward them, and physically interact with them through coordinated full-body motion. This perception-to-action loop, which humans execute effortlessly in everyday settings, requires a policy that maps high-dimensional visual input and language instructions to whole-body kinematic behavior.
We show that the missing link is a data-generation system that synthesizes paired visual observations within 3D Gaussian Splatting (3DGS) worlds and corresponding robot kinematic behaviors suitable for whole-body motion tracking. The key advantage of such a system is the decoupling it enables. We can prescribe the language commands, leverage privileged information in the simulated world (object poses, collision geometry, walkable regions) to ease the challenge of behavior generation, and render the corresponding egocentric observations as RGB images after the fact. By exploiting targeted annotation, oracle-conditioned motion synthesis, and hindsight rendering, we can scale up data production by orders of magnitude to generate synchronized language instructions, egocentric visual observations, and whole-body kinematic trajectories without any human demonstration effort.
Using this system, we synthesize 48,000 trajectories automatically within 600 GPU-hours in metric-scale indoor environments reconstructed by 3DGS. We then train a Vision-Language-Kinematics (VLK) policy that takes an egocentric image, a task instruction, and the current robot state as input, and predicts a short-horizon whole-body kinematic trajectory together with binary wrist-object contact labels. We treat these contact labels as auxiliary components of the kinematic state: they indicate whether each wrist should be in contact with the manipulated object and are readily available from our synthetic data. During deployment, a contact-aware whole-body tracker uses the predicted kinematics and contact labels to produce robot actions on the physical humanoid.
We evaluate the system on two fundamental task categories, navigation and object transport. Navigation requires the robot to move itself to a commanded location; object transport requires it to move an object to a commanded location. Together, they exercise the full perception-to-action loop: visual grounding, locomotion, picking up, carrying, and placing objects. Experiments in closed-loop simulation and on the physical G1 demonstrate that our synthesized interactions in reconstructed scenes provide effective supervision for sim-to-real perception-based humanoid loco-manipulation.
Synthetic Data Generation in Reconstructed Scenes
We generate training data by synthesizing Unitree G1 motions in metric-scale reconstructed scenes and rendering the corresponding egocentric observations. The pipeline has three stages: scene reconstruction and annotation, G1 motion synthesis in the annotated scenes, and egocentric rendering. Together, these stages produce paired vision-language-kinematics data.
We reconstruct indoor environments from Polycam scans captured with an iPhone 14 Pro. The scans combine RGB imagery with LiDAR-based depth, enabling metric-scale scene reconstruction. For each scene, we optimize a 3DGS representation that preserves the visual appearance and spatial layout of the real environment for egocentric rendering.
Vision-Language-Kinematics Policy
The generated vision-language-kinematics sequences provide paired supervision for learning a policy that maps robot-view observations and task instruction to future whole-body motion.
Experimental Setup
We use two physical environments: a lab-style scene and an apartment-style scene. For each environment, we create multiple physical layouts by changing the positions and orientations of selected furniture items, then scan each layout using the Polycam app on an iPhone 14 Pro and reconstruct it with 3D Gaussian Splatting (3DGS). This produces 4 lab and 4 apartment layouts, which are used for synthetic data generation and simulation evaluation. Real-world deployment is conducted in the same two physical environments, with furniture layouts manually varied across trials.
Synthetic Data Generation Details
Given a reconstructed scene with semantic 3D bounding boxes and walkable-region annotations, the pipeline samples task-relevant objects, feasible initial poses, waypoints, and task instructions; synthesizes G1-compatible trajectories; filters trajectories with scene penetrations; and renders paired egocentric observations. Each sequence contains synchronized egocentric observations, language instructions, and G1 whole-body kinematic trajectories.
Each environment contains 4 reconstructed layouts and 12 data-generation modes: six evaluated modes — Walk To, Turn Around, Pick (Floor), Put (Floor), Pick (Surface), and Put (Surface) — plus auxiliary modes for random walking, turning, carrying, and carry-turning. These auxiliary modes improve state coverage by exposing the policy to diverse walking and object-transport behaviors. We synthesize 1000 trajectories per layout and mode, yielding 48,000 trajectories per environment. On a single NVIDIA L40S GPU, synthesizing 1000 trajectories for one mode in one layout takes approximately 4 hours, while rendering the corresponding egocentric observations takes approximately 8.3 hours. These results show that, after scene reconstruction and annotation, the pipeline can automatically generate large volumes of paired VLK supervision, with generation parallelized across layouts and modes.
Effect of Visual Domain Randomization
We evaluate whether visual domain randomization during rendering improves robustness to visual perturbations in MuJoCo simulation (with IsaacSim rendering). Specifically, we evaluate walking-mode success in the lab scene under randomized lighting and camera conditions, and compare four training configurations: no randomization, lighting randomization only, camera randomization only, and full randomization. The full setting combines lighting variation and camera variation.
Scene Annotation
We annotate each reconstructed scene to support scene-aware motion synthesis. Since the 3DGS representation does not provide an explicit mesh or semantic labels, we first extract a point cloud from the reconstructed scene and use it as the geometric reference for annotation. We implement an interactive annotation tool using viser, which allows users to load the reconstructed 3D scene, inspect it in 3D, and add task-relevant annotations.
For semantic object annotation, we manually place oriented 3D bounding boxes around objects of interest and assign each box a semantic label, such as chair, table, or box. The annotation interface supports translating, scaling, and rotating each box to align it with the corresponding object in the scene. These semantic 3D boxes are used to sample target objects for navigation tasks and define task-relevant object locations for interaction synthesis.
We also annotate walkable regions in each scene. Users specify a set of points on the floor plane, and the tool connects these points to form polygonal regions. These regions define feasible areas for sampling humanoid initial poses and navigation waypoints. Together, the semantic object boxes and walkable regions provide the geometric and semantic structure needed for scene-aware motion generation.
Waypoint and Language Generation in 3D Scenes
We generate sparse waypoints from the annotated 3D scenes according to task-specific rules. For navigation tasks, we first sample a target object from the annotated semantic 3D bounding boxes. We then sample the initial humanoid pose and intermediate waypoints from the annotated walkable regions. The sampled poses are constrained to be collision-free and compatible with the task objective.
For box-interaction tasks, we generate waypoints and contact timing according to the object state and the height of the supporting surface. For picking motions, the waypoints guide the humanoid to approach the box, establish wrist-object contact, and lift the box from either the floor or a support surface. For placing motions, the waypoints guide the humanoid to carry the box toward the target support surface and release it at the desired location. Since lifting from the floor, lifting from a table, and placing onto surfaces require different arm heights and contact phases, we design the wrist contact frames and desired relative wrist poses based on the surface height. This task-specific design helps the synthesized motion maintain realistic contact timing, avoid abrupt vertical motion, and align the robot wrists with the box during lifting and placement.
The final waypoint sequence provides sparse spatial guidance for the motion synthesis model. These waypoints encourage the generated humanoid motion to follow feasible paths in the reconstructed scene.
Post-Processing for Generated Motion
Foot sliding and unrealistic hand-object contact are common artifacts in human motion synthesis and human-object interaction synthesis. To improve the quality of the generated motions, we apply lower-body post-processing following prior work. The key idea is to use the predicted foot-contact labels to identify contact onsets and contact phases. During each contact phase, we constrain the corresponding robot foot to remain fixed at its contact position by solving inverse kinematics.
To correct hand-object contact artifacts, we adapt the optimization approach from EgoAllo and introduce a wrist-pose matching term that drives the position and rotation of each wrist toward the input wrist pose expressed in the object's local frame. This encourages more realistic two-hand grasping contact during object interaction.
Domain Randomization in Rendering
To improve the real-world utility of the rendered data, we apply domain randomization during egocentric rendering. The randomization covers camera parameters, lighting, and image appearance. Camera randomization perturbs the camera extrinsics and focal length to account for small calibration mismatches between the virtual and physical ZED 2i camera setups. Lighting randomization changes the dome-light intensity and yaw rotation. We also apply image-space appearance augmentations, including perturbations to brightness, contrast, saturation, and hue, as well as Gaussian noise and Gaussian blur.
VLK Training Loss Details
We initialize the VLK policy from a pretrained vision-language model and fine-tune the full model on our generated vision-language-kinematics dataset. We adapt the action space of the pretrained model to our G1 kinematic representation.
The policy predicts the clean trajectory conditioned on the noisy trajectory, interpolation coefficient, current egocentric observation, task instruction, and current G1 kinematic state. The primary trajectory reconstruction loss is a mean squared error between the predicted and ground-truth joint positions and orientations over the trajectory horizon.
Metric for Evaluation in Simulation
While the RL tracking controller is trained in IsaacLab, we perform sim2sim evaluation in MuJoCo. During evaluation, the simulator is paused while the VLK policy performs trajectory inference at the end of each motion chunk, avoiding the need for real-time chunked execution. For each task mode, we hold out 10% of the synthesized trajectories (1000 trajectories per mode) for evaluation. The humanoid is initialized from the initial state of each held-out trajectory and evaluated on whether it can successfully complete the corresponding language-conditioned task.
For "Walk To" tasks, a rollout is considered successful if the humanoid reaches the specified target object or box, stops within 0.5 m of the target, and avoids colliding with or penetrating the target geometry.
For "Turn Around" tasks, success requires the humanoid to perform a clear turning motion toward the instructed direction, rather than remaining stationary or continuing forward motion.
For "Pick Up" tasks, the humanoid must successfully grasp a box placed either on the floor or on top of an object and maintain stable holding for at least 20 consecutive frames.
For "Put Down" tasks, success is defined as the humanoid reaching the specified target object and successfully placing the carried box onto the target surface or floor.
Motion Blur Handling During Deployment
We augment a subset of RGB observations with synthetic motion blur to improve robustness during real-world deployment where camera motion during locomotion naturally produces motion blur.
Conclusion
We presented a perception-based humanoid loco-manipulation system trained from synthesized interactions in reconstructed 3D scenes. Our pipeline generates Unitree G1-compatible navigation and box-manipulation trajectories, renders paired egocentric observations, and trains a VLK policy to predict whole-body kinematic trajectories for tracker-based execution. Experiments in simulation and on the physical Unitree G1 demonstrate object-directed navigation and box manipulation, showing that scene-grounded synthetic supervision can support sim-to-real humanoid loco-manipulation. These results suggest that synthesizing robot-compatible interactions in reconstructed scenes is a promising path toward scalable perception-conditioned humanoid control.
Our current interaction synthesis is limited by the coverage of OMOMO, which contains interactions with a limited set of large objects. As a result, the generated behaviors are better suited to bimanual transport of box-like objects than to grasping small objects such as cups or tools. Similarly, the contact-aware tracker stabilizes large-object manipulation through wrist-object contact, but does not address precise object grasping. Extending VLK to small-object manipulation would require richer interaction data and a low-level controller designed for precise object grasping.
Frequently Asked Questions
How is the synthetic training data generated without human demonstration? The pipeline uses scene reconstruction with 3D Gaussian Splatting, semantic annotation of objects and walkable regions, and oracle-conditioned motion synthesis to automatically generate 48,000 paired vision-language-kinematics trajectories in just 600 GPU-hours.
What tasks can the humanoid robot perform after training? The VLK policy enables two fundamental task categories: navigation (walking to a commanded location) and object transport (picking up, carrying, and placing boxes on floors or surfaces), tested both in simulation and on the physical Unitree G1 robot.
How does the system handle the sim-to-real gap for visual observations? Visual domain randomization during egocentric rendering perturbs camera parameters, lighting conditions, and image appearance (brightness, contrast, saturation, hue, noise, and blur) to improve robustness when deploying the policy in real-world environments.
What are the current limitations of this approach? The system is best suited for bimanual transport of box-like objects and does not handle grasping small objects such as cups or tools, which would require richer interaction data and a specialized low-level controller for precise grasping.