A new whole-body control framework called HANDOFF fuses three specialized expert policies — locomotion, whole-body manipulation, and fall recovery — into a single student controller that humanoid robots can use without retraining between tasks. By exposing a compact 10-dimensional task-space command interface, HANDOFF lets high-level planners drive diverse loco-manipulation behaviors using only wrist and base velocity references, replacing the need for full-body joint trajectories.
What the Researchers Built
The researchers built HANDOFF, a humanoid whole-body controller that separates task specification from joint-space execution. At the core is a 10-D task-space interface: it accepts 3-D pelvis-frame wrist position commands for each arm (total 6 dimensions) plus four velocity components — linear x/y, yaw rate, and torso height rate. This low-dimensional interface is expressive enough to encode simultaneous locomotion and manipulation but small enough for a high-level planner (like a large language model or finite-state machine) to drive without emitting dozens of joint angles.
To train the controller, the team used a mixture-of-experts distillation approach. They first trained three teacher policies: a whole-body control (WBC) teacher for precise manipulation and posture, a locomotion teacher for agile base tracking, and an AMP-based fall-recovery teacher. A single student policy learns from all three via context-based KL divergence, where a velocity gate blends the WBC and locomotion teachers for the body, while arm motions anchor to the WBC teacher. A recovery-masked term routes fall scenarios to the third expert.
Key Results
HANDOFF achieves velocity tracking performance comparable to state-of-the-art locomotion controllers while simultaneously enabling whole-body manipulation — a capability that prior single-teacher distillations could not cleanly produce. The paper reports:
- Body‐slice tracking matches the best single locomotion teacher on linear and angular velocity commands.
- Arm tracking preserves the WBC teacher’s ability to reach targets in the workspace without degrading base stability.
- Fall recovery successfully activates under large disturbances, restoring upright posture without task interruption.
In simulation and real-robot hardware rollouts, the controller was deployed in an untethered agentic stack where a vision-language model interpreted natural language commands into the 10-D interface. The robot performed tasks like walking to an object, reaching and grasping it, and carrying it — all without task-specific controller retraining. The controller’s compact interface also allows zero-shot transfer to new high-level planners.
How It Works
HANDOFF’s architecture is a two-stage pipeline: a high-level agentic planner produces 10-D commands, and a distilled student policy maps those commands to joint torques. The student is a mixture of three experts — WBC, locomotion, and fall recovery — whose contributions are gated by a learned router.
Teacher design - WBC teacher: Optimizes whole-body posture and arm end-effector tracking using a model-predictive control prior. It excels at manipulation but has poor velocity tracking. - Locomotion teacher: A reinforcement‑learning policy trained to track base velocity commands (linear x/y, yaw). It runs agile gaits but loses coordinated arm use. - Fall‑recovery teacher: An adversarial motion prior (AMP) policy that catches falls using rapid joint torques.
Student distillation The student is trained with a per-step context‑dependent KL loss. The router observes the current state and a context vector (velocity magnitude + recovery flag) to decide the blend: - Body slice: Convex blend of WBC and locomotion teachers, weighted by a learned gating network. - Arm slice: Anchored solely to the WBC teacher to preserve manipulation precision. - Recovery mask: On fall‑prone samples, the gate shifts entirely to the fall‑recovery expert via a binary AMP mask.
A subset‑aware load‑balancing loss prevents expert collapse: on non‑recovery samples the gate is renormalized over the WBC and locomotion experts and pushed toward uniform. The recovery expert is excluded from this term and separately supervised.
Deployment At runtime, the planner emits the 10‑D vector at 50 Hz. The student policy runs at 200 Hz and sends joint torques directly to actuators. A kinematic correction (Section 3.5) compensates for tool‑frame orientation errors when the interface’s 3‑D wrist target does not fully constrain the gripper pose.
No benchmark table is included because the paper provides relative comparisons (“matches state‑of‑the‑art”) without absolute numbers for velocity tracking error or manipulation success rate.
Why This Matters for Robotics
HANDOFF addresses a fundamental bottleneck in legged manipulation: decoupling high‑level planning from low‑level whole‑body control. Most humanoid controllers either treat the arms as fixed masses (sacrificing manipulation) or require planners to output joint‑space targets (impractical for general AI agents).
By exposing a 10‑D task‑space interface, HANDOFF enables: - Planner flexibility: Any planner that can compute wrist positions and base velocities — vision‑language models, classical motion planners, or hierarchical RL — can command the robot without knowing its joint limits or dynamics. - Zero‑shot task switching: The same controller handles walking, carrying, reaching, and fall recovery; no per‑task retraining or reward tuning. - Untethered deployment: The real‑robot stack runs entirely onboard, with the planner processing camera images and outputting 10‑D commands at 50 Hz.
For robot buyers and integration engineers, this means a humanoid from the browse humanoid robots on Robot Overflow could be programmed with a simple high‑level script rather than laborious motion planning per task. It brings legged manipulation closer to plug‑and‑play usability.
Limitations and Open Questions
The current interface exposes only 3‑D wrist positions per arm (pelvis‑frame), not full 6‑D gripper poses. This forces a runtime kinematic correction to handle orientation residuals, which can limit precision for tasks like peg‑in‑hole insertion. The authors note that extending to 6‑D tracking is future work.
Additionally, the motion‑data curation pipeline (Appendix C.1) requires offline filtering of trajectory clips using a center‑of‑pressure feasibility check. This step may be sensitive to simulation accuracy and could need retuning when porting to new robot morphologies. The fall‑recovery teacher also relies on a separate AMP prior whose quality depends on the motion dataset.
Finally, the agentic planner shown is just one implementation; the paper does not explore how performance degrades with planner latency or noisy commands. Robustness to sparse or intermittent command streams remains an open question.
Frequently Asked Questions
What exactly is the 10-D command interface? It consists of two 3-D pelvis-frame wrist target positions (one per arm) plus four base velocity components: linear x, linear y, yaw rate, and torso height rate.
Does HANDOFF require retraining for new tasks? No. The controller is task-agnostic; the same distilled policy handles locomotion, manipulation, and fall recovery. Only the high-level planner needs to be updated for new behaviors.
What robot hardware was used for real-world tests? The paper shows experiments on a full‑size humanoid robot with 12 actuated joints (6 per leg) and 7 degrees of freedom per arm, operating untethered.
How does HANDOFF compare to end-to-end whole-body control? Unlike end-to-end policies that directly map sensor inputs to joint torques, HANDOFF separates planning from control via the low-dimensional interface, making the controller composable with diverse planners and easier to debug.
Conclusion
HANDOFF demonstrates that a distilled mixture‑of‑experts student can unify locomotion, manipulation, and fall recovery under a compact task‑space interface. This design allows humanoid robots to switch between tasks without controller retraining, bringing us closer to practical legged manipulators that can follow abstract commands.