A Unitree G1 humanoid uses depth cameras and a pair of learned controllers to walk, run, climb stairs, and jump onto and off boxes in real environments. The system separates choosing a motion from controlling the robot’s body, then fine-tunes its motion generator with reinforcement learning; the researchers report improvements of up to 80 percentage points in motion selection and 25 points in task success.
What did the researchers build?
The researchers built a two-part control system for a humanoid robot. A motion generator looks at depth images and a requested travel speed, then creates a short whole-body movement plan. A separate tracking controller turns that plan into joint commands, adjusting the robot’s movements to follow the reference while staying balanced. The generator refreshes its plan every 0.24 seconds; the tracker runs continuously at 50 Hz.
The system was demonstrated on a Unitree G1 indoors and outdoors. It handled walking and running, standing, stairs, and box jumps. Rather than asking one policy to both interpret the scene and directly control every joint, the design assigns different jobs to the two policies: one chooses a motion that fits what the cameras see, and the other executes that motion.
To provide training examples, the team started with captured human movement and optimized it for the robot. That process produced hundreds of motion references for different speeds, directions, and terrain interactions. The resulting system connects motion data, terrain-aware planning, and real-time robot control in a single pipeline.
What were the key results?
The reported headline gains came from reinforcement-learning fine-tuning of the motion generator. Compared with the relevant pre-fine-tuning results, the approach improved motion selection by as much as 80 percentage points and task success by as much as 25 percentage points. These are maximum reported gains, not a promise that every task improves by the same amount.
| Reported measure | Improvement from fine-tuning |
|---|---|
| Motion selection | Up to 80 percentage points |
| Task success rate | Up to 25 percentage points |
The paper also reports real-robot demonstrations of walking, running, stair traversal, and box jumps, in indoor and outdoor settings. The system maintained commanded travel speed while adjusting its behavior around terrain, rather than requiring a new speed command whenever an obstacle appeared. Two depth cameras helped it see upcoming obstacles early enough to slow down. The researchers report that removing the upper camera severely impaired this capability.
These results show both a measurable fine-tuning benefit and a practical hardware demonstration. However, the available results do not give absolute success rates, a full task-by-task breakdown, or a direct numerical comparison with other robot systems. The percentage-point improvements should therefore be read alongside the qualitative evidence from the hardware tests.
How does the system work?
The process starts with motion references derived from human movement data. The team optimized those references using a robot dynamics simulator, which allowed the motions to better match the robot’s physical capabilities and target speeds. The optimization could also enforce periodic movement—for example, ensuring a walking cycle ends in a state that connects cleanly back to its start. The resulting library included 252 steady-state motions, such as walking in different directions and running, plus 46 references for stairs and box jumps. In total, the training library contained 10,000 clips spanning 140 terrain geometries.
The tracker learns to follow these references. Its feedback controller compares the robot’s current state with the planned movement and sends joint-position targets to the robot’s motor controllers. It tracks the positions of the robot’s body links, not just joint angles, and is trained to discourage unwanted contact, excessive torque, and movement beyond joint limits. Training also randomizes physical properties and sensor readings to make the controller less dependent on one exact simulated setup.
The motion generator is a transformer trained with flow matching, a method for learning how to produce a trajectory from a sequence of progressively refined outputs. It conditions its plan on depth images from two cameras, the requested velocity, and a compact record of its previous trajectory. The cameras provide direct depth input rather than a pre-built terrain map, and the system does not require odometry. At deployment, the generator uses eight refinement steps and takes around 11 milliseconds to run; the tracker takes less than 1 millisecond. The depth images are downsampled to 30 by 26 pixels for the tracker.
Why does this matter for robotics?
Humanoid robots need more than a reliable gait on a smooth floor. They must react to changes in the ground while coordinating their whole bodies, and they need to do so without waiting for a person to select a new movement at every obstacle. This system’s split between motion generation and motion tracking is a practical way to combine flexible planning with fast, frequent control.
The cameras also make the control loop more responsive to terrain. A higher, forward-facing camera can spot an obstacle before the robot reaches it, while a lower camera sees the nearby ground and legs. That lets the robot adjust its pace while keeping the original velocity command. For work in facilities with steps, raised surfaces, or changing floor layouts, early terrain awareness can help a robot adapt without requiring a mapped route.
The results are relevant to buyers evaluating humanoid robots on Robot Overflow, especially when a task involves mobility beyond flat-ground walking. The demonstrations are not proof of readiness for unsupervised commercial operation, but they show how perception, learned motion planning, and a fast stabilizing controller can work together on real hardware.
What are the limitations and open questions?
The system uses depth, which describes the shape and distance of objects but not what they mean. It can detect a box-shaped object without knowing whether that object is safe or appropriate to jump on. The paper does not describe a mechanism for expressing that kind of semantic instruction.
The training pipeline also still needs human input. People select or heuristically choose reference movements, and some transition settings require hand tuning. Fine-tuning relies on hand-designed rewards, which could be difficult to reuse for very different skills. The reported percentage-point gains lack a detailed task-by-task breakdown in the provided results, and the camera ablation is described qualitatively rather than with a numerical measure. Further testing would be needed to understand reliability across longer deployments, varied environments, and tasks outside the demonstrated skill set.
What questions do buyers and engineers ask?
Does the robot plan and control its joints with the same policy? No. A generator creates short motion plans, and a faster tracking controller converts them into joint targets.
What sensors does the system use? It uses two depth cameras, one positioned high on the torso and another lower down. The policies take in depth images directly and do not require odometry.
Which movements were demonstrated? The Unitree G1 walked, ran, stood, traversed stairs, and jumped onto and off boxes in indoor and outdoor demonstrations.
Are the reported improvements absolute success rates? No. The paper reports gains of up to 80 percentage points in motion selection and 25 points in task success, but those figures are not absolute success rates.
What is the takeaway?
This work shows a humanoid robot using depth perception to select movements for varied terrain while a separate controller tracks those movements on hardware. Its fine-tuning results are promising, while semantic understanding, reduced manual setup, and more detailed performance measurements remain important steps toward broader deployment.