Caltech researchers built PAC-MAN, a perception-aware safety framework that taught a Unitree G1 humanoid to dodge thrown balls using only its onboard depth camera and joint sensors. The key discovery: the strongest safety guarantees only pay off when the robot's perception is good enough to support them, so lighter safety structures win on real hardware.
What the Researchers Built
PAC-MAN is a training framework that fuses control barrier functions (CBFs) with reinforcement learning so a humanoid can dodge an incoming ball with its entire body. A ball launches toward the robot, and a trial succeeds only if no body link — not even a finger or ankle — makes contact and the robot keeps its balance. There is no reference motion to follow; the robot generates the evasion on its own from raw sensor signals.
The framework tests two levels of safety structure. Link-CBF is a lightweight per-link penalty used during training to steer the policy away from contact on any body part. Joint-CBF is a stronger whole-body projection of the safe set that can guide training or act as a runtime safety filter when accurate ball state is available.
At deployment, the robot sees only what onboard sensing provides: proprioception plus a depth image masked down to a segmented ball. An adversarial motion prior shapes evasions into crouches, leans, sidesteps, and leaps rather than stiff contortions. Two deployment conditions are studied: a single calibrated throw and a continuous loop where the robot walks back to its station and recovers between throws.
Key Results
PAC-MAN's central finding is a coupling between safety structure and perception: stronger barrier structures only pay off when the observations support them.
With a state oracle feeding exact ball positions, the full Joint-CBF with runtime filtering was the strongest configuration. Under a realistic fixed onboard camera, however, the lighter Link-CBF — which the policy internalizes during training and needs no runtime filter — was the best deployable option. An oracle-aimed gimbal in simulation sat between the two, showing that keeping the threat in view longer unlocks more safety structure.
The team deployed the fixed-camera Link-CBF policy zero-shot on a Unitree G1 humanoid using only onboard depth and proprioception. The robot successfully evaded live throws, and because the ball is segmented semantically in RGB, the same policy handles different balls without retraining.
The benchmark also uses a strict any-link contact criterion. The results show many apparent dodges fail through limb grazes, so scoring by pelvis distance alone overstates safety — a caution for whole-body robot evaluation.
How It Works
Control barrier functions encode safe state sets and automatically correct commands that would push the system into danger — but only if the relevant state is available. Onboard perception breaks that assumption: a ball may be visible only briefly, leave the field of view, or register as a few noisy depth pixels. PAC-MAN treats the threat representation and the barrier design as one coupled safety problem rather than independent modules.
During training, CBF structure enters the policy in two ways. Link-CBF adds a per-link penalty to the reward, teaching the policy that any body part can be hit. Joint-CBF projects the safe set into joint space across the whole body, providing a stronger signal — and can be enforced at test time as a privileged runtime filter when accurate ball state exists.
Deployment observations are deliberately sparse: proprioception plus a depth image masked to a compact ball-only view, produced by segmenting the ball in RGB and gating the depth channel. The fixed camera sees the ball for a short window, which limits how much barrier structure the policy can internalize. An adversarial motion prior (AMP) regularizes the motion so evasions emerge as crouches, leans, sidesteps, and leaps.
Failures are scored stringently. Any link contact counts as a hit, and a fall is declared if torso orientation, base height, or sustained crouch crosses a threshold. The benchmark calibrates on-target throws against a frozen statue and reports both reset success and the deployment loop.
Why This Matters for Robotics
Humanoid robots moving into warehouses, factories, and public spaces will face transient hazards that no precomputed path can avoid: a falling box from a shelf, a pallet swinging off a forklift, a coworker stepping into the robot's path. PAC-MAN demonstrates a whole-body reactive reflex that runs from onboard sensing alone — no external motion capture, no infrastructure, no teleoperation.
The deeper lesson applies to any safety-critical robot: the safety filter is only as good as the perception feeding it. Designing a strong barrier and then assuming clean state will fail in the field; matching the barrier to what the sensors can actually deliver is the engineering that matters. That is why browse humanoid robots on Robot Overflow and warehouse robots increasingly need perception-aware control stacks rather than textbook safety layers.
For buyers, this points toward humanoids whose safety behavior is learned end-to-end from real sensor data — and validated under contact-based, whole-body metrics rather than simple distance thresholds.
Limitations and Open Questions
The active gimbal that restored Joint-CBF usefulness was oracle-aimed in simulation; a deployable tracker-aimed version needs additional hardware and remains future work. Physical validation ran with the fixed camera only, and the ball-tracking pipeline depends on RGB segmentation, which is harder in low light or visually cluttered environments. The benchmark targets a single ball per trial, while real workplaces present multiple simultaneous dynamic threats. And although the zero-shot deployment succeeded, sim-to-real gaps persist. Releasing the benchmark and training pipeline will let the community quantify those gaps and stress the framework under new ball types, speeds, and robot platforms.
Frequently Asked Questions
What is a control barrier function (CBF) in plain English? A CBF defines a safe zone in the robot's state space and automatically corrects any command that would push the robot out of it. It provides a mathematical guarantee that the robot stays inside the safe set — provided the state is known.
Why did the simpler Link-CBF beat the stronger Joint-CBF on the real robot? Joint-CBF needs accurate ball state to enforce its whole-body projection at runtime, which a fixed onboard camera cannot reliably provide. Link-CBF builds safety into the policy during training, so it works with the sparse observations the robot actually has.
Does this work only for dodgeball? No. Dodgeball is a deliberately hard testbed for transient, whole-body collision avoidance. The same perception-aware safety structure applies to any robot that must react to unexpected moving objects — falling stock, swinging loads, or people stepping too close.
Did the robot use motion capture or external tracking during the real deployment? No. On hardware, the policy used only onboard depth and proprioception, with the ball segmented semantically from RGB. No external infrastructure was involved.
Conclusion
PAC-MAN shows that whole-body collision avoidance for humanoids is a perception problem as much as a control problem. By matching the safety barrier to what onboard sensors can actually deliver, the framework produces a robot that dodges real throws zero-shot. The result is a practical blueprint for reactive safety in humanoids heading into human environments.