MemoryVLA++ extends standard vision-language-action (VLA) models with a dedicated memory bank and a world model that imagines future visual states, enabling robots to handle tasks requiring temporal reasoning, memory recall, and predictive planning. By combining past observations with imagined future scenarios, the model significantly improves success rates on long-horizon and memory-dependent manipulation tasks across simulation and real-world robots.
What the Researchers Built
The team developed MemoryVLA++, a vision-language-action model that adds two critical components missing from prior VLA approaches: a temporal memory module and an imagination module (a world model). The memory module stores compressed latent representations of past observations over a variable-length horizon (from 16 to 256 steps), allowing the model to reason about events that happened earlier in a task. The imagination module — a diffusion-based world model trained separately — generates predicted future visual features given current observations and action plans, letting the robot “imagine” outcomes before executing.
The model outputs continuous 7-DoF actions (end-effector translation, rotation, and gripper state) for both single-arm and dual-arm manipulation. It was tested on three different robot platforms: Franka Panda, WidowX, and a custom dual-arm setup. Across 5 simulation benchmarks (Libero, SimplerEnv, Mikasa-Robo, Calvin, Libero-Plus) and 17 real-world tasks, MemoryVLA++ consistently outperformed the strong CogACT baseline, especially on long-horizon and memory-dependent tasks like sequential button pushing or packing bags on a conveyor belt.
Key Results
Evaluations covered nearly 200 tasks with extensive variations. On the Libero benchmark (130 tasks), MemoryVLA++ achieved higher success rates than CogACT across all five suites — Spatial, Object, Goal, Long-10, and Long-90 — with the largest gains on the long-horizon Long-90 suite. On SimplerEnv, trained on BridgeData-v2 and tested zero-shot, the model outperformed prior methods by a clear margin in general manipulation success.
On Mikasa-Robo (5 memory-dependent tasks), MemoryVLA++ mastered tasks requiring recall of past object states, where standard VLA models often failed on the second or third step. In the Calvin benchmark (long-horizon language-conditioned chain tasks), the model completed more than 4 out of 5 consecutive instructions on average, surpassing the previous best by over 10 percentage points. On Libero-Plus (robustness and generalization with 7 out-of-distribution variations), the model showed strong robustness even in zero-shot settings, and further improved with supervised fine-tuning.
Real-world results from 17 tasks — including “Guess Where” (memory-dependent) and “Conveyor Scan-Pick” (imagination-dependent) — demonstrated that MemoryVLA++ could handle novel object arrangements, variable conveyor speeds, and interrupted sequences. For example, on “Bag Pack & Zip”, the dual-arm robot reached into the bag, packed five objects, and zipped the bag with high partial-score success across 10 consecutive trials.
How It Works
MemoryVLA++ follows a modular architecture built on a pretrained VLA backbone. The key innovation is adding a temporal-aware action expert that processes three streams of information: 1) current visual observations (image tokens), 2) a sequence of past latent memories, and 3) imagined future visual features.
Memory module: During training, the model compresses each observation-action pair into a compact latent token via an encoder. These tokens are stored in a ring buffer. At inference, the last _N_ memory tokens are retrieved and fed through cross-attention with the current observation tokens, producing a context-aware feature.
World model: A separate diffusion model is trained (40k steps in simulation, 20k on real data) to generate future visual feature tokens given current observations and an action plan. During long-horizon tasks, the robot repeatedly imagines the next few steps, and the imagined features are combined with real observations to select actions.
Temporal-action fusion: The cognition-aware features from memory and imagination are then passed through a perception-attention layer that injects fine-grained visual details from the current frame, followed by an FFN. The final denoised representations are projected by an MLP to output 7-DoF continuous actions (translation, rotation, gripper). Training uses mean squared error (MSE) loss between predicted and target action sequences.
The model supports variable memory lengths — 16 steps for general tasks, 256 steps for long-horizon temporal tasks — and can operate at 5–10 Hz on a single GPU, making it practical for real-time robot control.
Why This Matters for Robotics
Adding explicit memory and imagination to VLA models addresses a critical bottleneck in deploying robots for real-world tasks that require more than reactive picking and placing. Warehouse robots that must remember which items have been scanned or kitchen robots that need to follow a sequence of steps without resetting each time are prime candidates for this approach.
For example, a warehouse robot tasked with sorting mixed items onto a conveyor could use the imagination module to anticipate collisions and plan grasps ahead of time. Similarly, humanoid robots performing multi-step assembly would benefit from memory of previous actions. MemoryVLA++ also shows promise for collaborative dual-arm tasks — a capability that is still rare in commercial VLA systems.
The modular design means existing VLA models (like CogACT or Octo) can be retrofitted with temporal modules without full retraining, lowering the barrier for adoption by robotics integrators. As the world model improves, robots can increasingly operate in dynamic, partially observable environments rather than fixed, scripted cells.
Limitations and Open Questions
While MemoryVLA++ achieves strong results, several limitations remain. The world model is trained offline on collected demonstrations and may not generalise to truly novel dynamics — for example, objects with unknown mass or friction. The memory buffer stores latent tokens, which compress information but can lose fine-grained details needed for precise tasks (e.g., inserting a peg into a tight hole). Computational cost scales linearly with memory length, making real-time operation on embedded hardware challenging. Finally, the imagination module is not jointly trained with the action policy; integrating them end-to-end could yield further gains.
Frequently Asked Questions
What is a VLA model? A Vision-Language-Action model takes camera images and natural language instructions as input and directly outputs robot motor commands, without needing explicit motion planning or perception pipelines.
How does the memory module differ from simply using a frame history? MemoryVLA++ stores compressed latent representations rather than raw pixels, which reduces computation and allows the model to retain longer sequences (up to 256 steps) without memory overflow.
Can MemoryVLA++ be used on any robot? Yes, the output action space (7-DoF continuous) is common to most articulated robot arms. The model has been validated on Franka, WidowX, and a dual-arm setup, and can be adapted to other platforms with consistent action definitions.
What hardware is needed to run MemoryVLA++ in real time? The authors report real-time inference (5–10 Hz) on a single Nvidia RTX 4090 GPU. On lighter hardware, the memory length or world model resolution may need to be reduced.
Conclusion
MemoryVLA++ introduces a practical way to give VLA-based robots temporal awareness through a memory bank and a world model. Tested across nearly 200 tasks in simulation and the real world, it consistently outperforms baseline models, especially on long-horizon and memory-dependent challenges. For robotics teams looking to move beyond reactive pick-and-place, this approach offers a clear upgrade path.