HEFT is a new teleoperation framework that lets a full-size humanoid robot carry up to 24 kilograms in both hands while following complex whole-body motions commanded by a human operator wearing VR gear. The system solves two long-standing challenges in humanoid teleoperation: noisy operator inputs and variable payload difficulty that changes with body posture.
What the Researchers Built
The team built HEFT (Heavy-payload Full-size Humanoid Teleoperation), a control framework that runs on L7, a 175 cm, 65 kg humanoid robot with 29 actuated joints. The system accepts real-time commands from a human wearing VR controllers, then drives the robot to copy the motion while carrying heavy loads.
Two key innovations make this possible. First, Privileged Motion Guidance (PMG) lets the robot learn from rough VR inputs during training by comparing them against a clean, physically plausible reference motion that the teacher policy has access to—but the deployed student policy never needs. Second, Windowed Payload Curriculum (WPC) breaks a motion sequence into short windows and, for each window, uses an expert simulator to determine the maximum safe payload. The robot’s controlling policy is then trained on loads that vary window by window, so it learns to handle different weights at different phases of a movement without a task-specific carrying controller.
The result is a single policy that can perform loaded locomotion—turns, forward/backward walking, squats—with real two-hand payloads up to 24 kg, all while being driven by a human operator through off-the-shelf VR equipment.
Key Results
HEFT was evaluated in both simulation and on the physical L7 hardware. On the hardware, the robot successfully performed dynamic motions—including turns, forward/backward locomotion, and squats—while carrying two-hand payloads up to 24 kg. This is a significant achievement for a full-size humanoid, as most prior work either handles much lighter loads or uses task-specific controllers that cannot generalise across motion types.
In simulation, the team tested on both their L7 platform and a smaller G1 robot to compare against earlier approaches. Key findings:
- PMG reduces raw-reference drift: When the operator’s VR tracking drifts (e.g., because of occlusion or calibration errors), the PMG-trained policy tracks the intended motion much more closely than a policy trained directly on raw VR references.
- WPC expands high-payload capability: Compared to a global fixed payload cap, the windowed curriculum increases the maximum feasible payload—the policy trained with WPC can carry heavier loads during motion phases that allow it, while still performing well on unloaded motions.
- Single-policy efficiency: HEFT does not need separate controllers for carrying vs. non-carrying tasks; one trained policy handles both unloaded and heavily loaded operations.
Quantitative comparisons in simulation showed that HEFT outperformed both a baseline trained without PMG (higher tracking error) and one trained with a uniform payload schedule (lower success rate on heavy loads).
How It Works
HEFT uses a teacher-student training architecture. During training, a teacher policy has access to privileged information: a physically reconstructed version of the VR operator’s motion (not the noisy raw data), plus the expert-computed payload caps for each time window, plus the true payload force applied to the wrists. The teacher learns to map raw VR inputs directly to joint torques, but it does so guided by the clean reference.
Then a student policy is trained to mimic the teacher’s behaviour without those privileges. An adapter network predicts the teacher’s latent state from the student’s inputs alone. At deployment, only the student and adapter run on the robot—no reconstruction, no payload caps, no simulator.
Payload curriculum in detail: The expert (a reference-tracking policy with full state information) rolls out the robot under increasing downward forces on the wrists, in 5 kg steps, testing from 30 kg downward. For each short window—say, during a turn or squat—the expert finds the largest load the robot can physically sustain without falling. These caps are then used to train the student: loads are sampled per window within that cap, with fractions split randomly between the two wrists and applied within a 12° cone around vertical.
The robot’s proprioceptive inputs (joint positions, velocities, IMU, etc.) and raw VR references are fed into both teacher and student. The teacher additionally conditions on the reconstructed reference, window caps, and payload state. The student depends only on its own latent (provided by the adapter) and the raw VR inputs. Training proceeds in three stages: teacher PPO optimization under PMG and WPC, latent distillation into the adapter, and fine-tuning the student while rolling out the adapter-predicted latent.
Why This Matters for Robotics
This work directly addresses a critical gap in humanoid robotics: the ability to teleoperate a full-size robot doing useful work while carrying heavy objects. Most current humanoid teleoperation systems either require the operator to hold lightweight tools or rely on task-specific controllers that cannot handle varied motions. HEFT shows that a single learned policy can manage the whole-body coordination needed for loaded walking, turning, and squatting.
For warehouse, construction, or disaster-response applications, this is a practical step toward deploying humanoid robots that can be remotely controlled to lift and move boxes, equipment, or debris. The approach also means that operators can use low-cost VR headsets without needing expensive motion-capture suits, and the system still performs reliably.
By handling payloads up to 24 kg—roughly the weight of a large display case or a heavy toolbox—HEFT demonstrates that humanoids can be more than just dexterous manipulators; they can be mobile load-carriers. This opens the door for tasks where a used cobot on a fixed base would be insufficient, and where teleoperation gives the human in the loop the high-level decision-making.
Limitations and Open Questions
HEFT still requires offline processing: the VR reference motions must be reconstructed physically, and the expert must label payload caps—both steps that take time and computation. Transferring the method to a new robot or a different tracker setup means re-running this data-preparation pipeline.
The payload model is simplified: it only applies vertical forces to the wrists and does not model gripping, object shape, sliding, or contacts with the environment. In real tasks, a box might be hard to grasp or might lean against surfaces.
Hardware validation was limited to one full-size platform (L7). Testing on other humanoids would reveal how well the framework generalises. Finally, the approach has not yet been demonstrated in unstructured outdoor environments or with unseen object geometries.
Frequently Asked Questions
How much weight can HEFT handle? Up to 24 kg held in both hands, distributed between the wrists, across motions like walking, turning, and squatting.
Does the human operator need special equipment? No, just standard VR controllers. The system learns to handle the noise in raw VR tracking without requiring expensive motion capture suits.
Is HEFT ready for commercial use? The framework has been validated on one robot platform in a lab setting. Commercial deployment would require further testing on different robots and environments, plus integration with safety systems.
Can HEFT work without any payload curriculum? It could, but performance on heavy loads would degrade. The windowed curriculum is what allows the robot to carry more weight during motion phases that permit it while preserving agility during others.
Conclusion
HEFT proves that a single policy can enable a full-size humanoid to copy a human’s whole-body motions while carrying substantial payloads—up to 24 kg. The combination of privileged motion guidance and a motion-adaptive payload curriculum makes it possible without task-specific controllers or expensive hardware.