In memory-dependent tasks, long-horizon visual history is not necessarily required as a direct input to the execution model at every control step. This history is primarily needed to determine the next segment-level plan and the desired visual evolution, while execution can operate by following a memory-grounded plan. This motivates MaP-WAM, which maintains long-term memory as multimodal episodic context and uses it as planning-time evidence to generate compact plans. By decoupling memory-grounded planning from plan-conditioned execution, MaP-WAM keeps the executor context length fixed and reduces execution-time latency while preserving fine-grained grounding in long-horizon visual memory.
MaP-WAM achieves 83.3% and 78.0% success rates on RMBench and real-robot tasks, respectively, while maintaining approximately constant executor inference latency as task history grows.
Contributions
Memory-as-Plans Framework
We propose MaP-WAM, which converts long-horizon episodic evidence into memory-grounded plans, enabling fixed-context execution while retaining visual grounding.
Memory-Grounded Planning
We introduce a causal world model that translates long-term visual context and the predicted segment-level language plan into a visual plan as fine-grained execution guidance.
Progress-Aware Execution
We introduce WAP, which jointly models visual dynamics, action chunks, and execution progress, and combines MoT-based progress prediction with plan-observation alignment to enable variable-duration execution and adaptive segment transitions.
Conclusion
We presented MaP-WAM for memory-dependent robotic manipulation, where task execution requires information beyond the current observation. Rather than repeatedly conditioning the executor on dense visual histories, MaP-WAM converts structured episodic context into memory-grounded plans that couple task semantics with anticipated visual evolution.
By combining memory-grounded planning with progress-aware execution, MaP-WAM closes the loop among planning, execution, and context updates while preserving planning-time access to long-term, fine-grained visual evidence and enabling efficient KV-cached inference with a fixed executor context.
Experiments in simulation and on real robots show that MaP-WAM improves performance on long-horizon memory-dependent tasks while maintaining approximately constant per-chunk executor latency as task history grows.
Limitations
MaP-WAM currently builds on the segment structure available in existing benchmarks to organize memory and progress. Extending the planning to unsegmented demonstrations via automatic segment discovery is a natural next step.
Additionally, the plan-observation alignment mechanism adopts a lightweight, training-free matching metric, and learned similarity measures may further improve calibration robustness in visually complex scenes.
Frequently Asked Questions
What is MaP-WAM designed for? MaP-WAM is designed for memory-dependent robotic manipulation tasks that require information beyond the current observation.
How does MaP-WAM use long-term visual history? It uses long-term multimodal episodic context as planning-time evidence to generate compact, memory-grounded plans.
What is the role of progress-aware execution? Progress-aware execution models visual dynamics, action chunks, and execution progress to support variable-duration execution and adaptive segment transitions.
What success rates does MaP-WAM achieve? It achieves 83.3% success on RMBench and 78.0% on real-robot tasks.