Researchers have developed HarmoWAM, a robotic manipulation system that uses a shared world model with two action experts—one for generalizable motion and one for precise interaction. An adaptive gating mechanism switches between them based on real-time visual feedback, achieving zero-shot generalization in new environments without sacrificing fine-grained manipulation accuracy. This approach outperforms prior world action models across multiple tasks and out-of-domain scenarios.
What the Researchers Built
HarmoWAM is an end-to-end World Action Model (WAM) that unifies two complementary control strategies under a single learned world model. Previous WAMs could do one or the other—either generalize to new environments with coarse motion, or execute precise interactions in familiar settings—but not both. The HarmoWAM architecture solves this by deploying two parallel action experts: a predictive expert optimized for long-horizon, coarse transit motion (e.g., moving an arm toward a target across the workspace), and a reactive expert designed for short-horizon, contact-rich interaction (e.g., grasping, inserting, aligning).
The system’s core innovation is a Process-Adaptive Gating Mechanism that automatically discriminates between transit and interaction phases using visual features from the shared world model. During execution, the gating classifier routes control between the two experts at every timestep with over 96% accuracy. The world model itself provides spatio-temporal latent features that condition both experts, giving them a consistent understanding of scene dynamics, object relationships, and task progress.
The system was trained on six tasks spanning single-arm and bimanual manipulation, including pick-and-place, stacking, pouring, and insertion. It was then tested in three types of out-of-domain (OOD) scenarios: unseen backgrounds, unseen object positions, and unseen objects.
Key Results
- Gating classifier accuracy: 96.95% on held-out test demonstrations (1,637 frame pairs).
- OOD performance: In the most challenging position OOD scenario, alternative routing methods like simple averaging caused a 46% drop in task success, while Keyframe-Based Averaging reduced the drop to 31%. HarmoWAM’s Process-Adaptive Gating significantly outperformed both.
- World model latent features are critical: Removing these features from the reactive expert dropped its in-distribution (ID) performance to 65% and OOD average to 54%. Removing them from the predictive expert dropped ID performance from 95% to 62%.
- Successful generalization across three OOD dimensions: background clutter and lighting, unseen spatial positions, and novel objects with different shapes and appearances (e.g., carrot → pepper, Coke can → Red Bull can).
The following table summarises the ablation study across the Put Flowers in Vase and Pick Fruit to Plate tasks (average performance reported):
| Variant | ID Performance | OOD Position | OOD Background | OOD Object |
|---|---|---|---|---|
| Averaging routing | Baseline | −46% drop | −38% drop | −42% drop |
| Keyframe-Based Averaging | Near baseline | −31% drop | −28% drop | −33% drop |
| HarmoWAM (Process-Adaptive Gating) | 95% | Small drop | Small drop | Small drop |
| HarmoWAM without world model latent features (reactive expert) | 65% | 54% avg OOD | — | — |
| HarmoWAM without world model latent features (predictive expert) | 62% | — | — | — |
Numbers are approximate from ablation tables; exact values are available in the full paper.
How It Works
HarmoWAM is built on a single shared world model that outputs video-like latent representations of the scene at each timestep. These latents encode both current observations and predictions about future states, giving the system a temporally coherent understanding of the world.
The world model’s latent features are fed into two parallel action heads: - Predictive expert: Outputs open-loop, long-horizon action sequences (e.g., a trajectory to approach a target). It’s trained to minimize errors over multiple timesteps, which forces it to learn generalizable motion patterns. - Reactive expert: Outputs closed-loop, short-horizon actions conditioned on immediate visual feedback. It’s trained with fine-grained contact awareness and reacts to subtle changes in the scene.
The Process-Adaptive Gating Mechanism is the key enabler. During training, the system automatically labels each timestep as either “transit” or “interaction” using a simple heuristic: a 20-frame window before and after any keyframe (e.g., when gripper position deviates from a straight line by more than a threshold). For dual-arm tasks, if either arm meets the interaction criterion, the timestep is labeled as interaction. A classifier is trained on these labels using the world model latents as input.
At inference, the gating classifier runs at every timestep and activates one expert. The predictive expert drives the robot during transit phases (coarse, high-speed motion), and the reactive expert takes over during interaction phases (contact establishment, fine adjustment, release). The switch is seamless because both experts are conditioned on the same latent features from the same world model, ensuring no state discontinuities.
The action space is 7-dimensional for single-arm tasks and 14-dimensional for dual-arm tasks. Training uses a weighted sum of Smooth L1 distances between predicted and ground-truth actions, with additional auxiliary losses from the world model.
Why This Matters for Robotics
Most current manipulation systems either excel at generalizable motion (like navigation or pick-and-place from varied poses) or at precise interaction (like peg-in-hole insertion or assembly), but not both. This forces system integrators to choose between flexibility and accuracy—or to deploy separate control stacks that are difficult to coordinate.
HarmoWAM demonstrates that a single unified policy can handle both phases using a computationally lightweight gating mechanism. For real-world applications, this means: - Industrial robots performing bin-picking and assembly can adapt to new parts and positions without manual retuning. - used cobots for sale could be equipped with such a system to handle both order picking (transit) and precise placement (interaction) in warehouse settings. - warehouse robots needing to cope with changing layouts, clutter, and novel objects would benefit from the zero-shot generalization shown in the OOD tests.
The approach is especially promising for bimanual tasks where one arm may be in transit while the other is in interaction—the gating mechanism handles both arms simultaneously. This could unlock more capable humanoid robots that need to coordinate arms for tasks like lifting, carrying, and assembling.
Limitations and Open Questions
- Task scope: The six tasks, though diverse, are still laboratory demonstrations. The system has not been tested on highly deformable objects (cloth, cables) or tasks requiring tool use.
- OOD robustness: The gating classifier achieved 96.95% accuracy on held-out demonstrations, but in completely unseen environments the accuracy may drop. The paper doesn’t report OOD accuracy for the gating itself.
- Real-time constraints: The full pipeline includes a world model and two action heads. Latency on edge hardware could be a bottleneck for high-speed assembly.
- Scalability: Training requires a world model that generalizes—currently an open challenge. If the world model fails to produce meaningful latents for a novel scene, both experts degrade.
- Safety in transit: The predictive expert operates open-loop. If a dynamic obstacle enters the workspace, the system may not react quickly enough until it switches to the reactive expert.
Future work could explore end-to-end learning of the gating policy, handling longer horizons, and applying the paradigm to mobile manipulation or locomotion.
Frequently Asked Questions
What problem does HarmoWAM solve? It solves the trade-off between coarse, generalizable motion and precise, fine-grained manipulation in robot learning. Prior world action models could do one well but not both. HarmoWAM uses a gating mechanism to seamlessly combine a predictive expert for transit and a reactive expert for interaction.
How does the gating mechanism know when to switch experts? During training, the system automatically labels frames as “transit” or “interaction” using a temporal window around keyframes (e.g., when gripper velocity changes sharply). A classifier is trained on these labels using features from the shared world model. At inference, it runs at every timestep and switches experts accordingly.
Does HarmoWAM work for bimanual robots? Yes. For dual-arm tasks, if either arm enters an interaction phase, the entire timestep is labeled as interaction. The gating mechanism then switches both arms to the reactive expert, ensuring coordinated fine control.
What hardware was used for experiments? The paper does not specify the exact robot platform, but the action space (7 or 14 dimensions) suggests a standard 7-DOF robot arm with a parallel gripper, possibly used in bimanual configuration.
How well does it generalize to new objects? In the “unseen object” OOD scenario, HarmoWAM successfully manipulated substitutes like a pepper for a carrot and a Red Bull can for a Coke can, adapting grasping strategies and interaction patterns accordingly.
Is the entire system trained end-to-end? Yes. The world model, both action experts, and the gating classifier are all trained jointly using a weighted sum of losses. The world model provides latent features that condition both experts, ensuring consistency.
Conclusion
HarmoWAM shows that a single world model can support both generalizable transit motion and precise manipulation by routing control between two complementary action experts through an adaptive gating mechanism. With a gating accuracy above 96% and strong performance across three out-of-domain scenarios, this work points toward robot control systems that no longer sacrifice generality for precision—or vice versa.