Recent vision-language-action (VLA) models struggle to keep track of context during long manipulation sequences. Researchers have built a dual memory system—a short-term buffer and a long-term store—that lets a robot arm remember what it did earlier and what it needs to do next, boosting success rates on complex, multi-step tasks.
What the Researchers Built
Most current VLA models compress a history of observations into a single fixed-size embedding. That works fine for pick-and-place, but fails on long tasks like assembling a toy car, where the robot needs to recall earlier steps. The team introduced a Dual Latent Memory architecture that separates memory into two compartments: a short-term latent memory that tracks fine-grained recent actions, and a long-term latent memory that stores high-level task progress.
The model encodes each video frame and language instruction into these two memory banks. The short-term memory updates every step, while the long-term memory compresses information across episodes. During action generation, the policy attends to both memories, allowing it to adapt its behaviour based on how far into the task it is. The result is a VLA model that can handle tasks with up to 10 steps without degrading.
The researchers tested their system on a simulated tabletop environment with single-arm tasks: stacking blocks, opening drawers, and assembling a toy car. They compared against a baseline VLA without explicit memory and against a stacked-LSTM approach.
Key Results
The dual memory model outperformed all baselines on long-horizon tasks. On the 10-step toy car assembly, the baseline VLA succeeded only 12% of the time. The dual memory model succeeded 82% of the time—a nearly 7× improvement.
On medium-length tasks (5 steps), the dual memory model scored 94% success versus the baseline’s 63%. Even on short 3-step tasks, the improvement was noticeable: 98% vs 90%.
The following table shows the success rates across different task lengths:
| Task Length | Baseline VLA | Dual Latent Memory | Improvement |
|---|---|---|---|
| 3 steps | 90% | 98% | +8% |
| 5 steps | 63% | 94% | +31% |
| 10 steps | 12% | 82% | +70% |
The model also generalised to unseen task variations—e.g., different object colours or slightly different drawer heights—without retraining, maintaining a 78% success rate on out-of-distribution 5-step tasks.
How It Works
At each time step, the robot receives an RGB image and a natural-language instruction (e.g., “pick up the red block and place it on the green plate”). A vision encoder (a pretrained ViT) extracts image features, and a language encoder (CLIP) extracts text features.
The short-term memory is a recurrent state—a GRU that updates after every action. It stores the last few (typically 4–8) observations and actions, giving the policy fine-grained temporal context.
The long-term memory is a fixed-size latent vector that gets compressed and stored at the end of each subtask. The model learns to segment the task into subtasks using a learned attention mask over the action history. Only when a subtask is complete (e.g., the gripper has released a block) does the long-term memory update. This prevents unnecessary forgetting of high-level progress.
During action prediction, the policy (a small transformer) takes the current observation, the short-term memory state, and the long-term memory state, and outputs a 7-DoF action (3D translation, 3D rotation, 1D gripper). The memory update is trained end-to-end with behavior cloning on demonstration data.
The dual memory adds only about 2.3 million extra parameters compared to a baseline VLA, making it practical for real-time deployment on a single GPU.
Why This Matters for Robotics
Long-horizon manipulation is the bottleneck that keeps many robot arms out of messy real-world environments. A system that can remember what it has already done without expensive external state tracking or repeated instructions is a huge step towards practical home and warehouse assistants.
For operations managers evaluating used industrial robots, this research shows that even existing robot hardware can handle more complex tasks if the control software gains better memory. Similarly, humanoid robots with bimanual dexterity would benefit from this approach when performing multi-step assembly.
The dual memory approach also requires no change to the robot’s physical setup—purely an algorithmic upgrade. That means integrators could retrofit existing warehouse robots and pick-and-place arms with better long-horizon capabilities.
Limitations and Open Questions
The experiments were conducted in simulation; real-world deployment will face visual noise, lighting changes, and sensor drift that could affect memory fidelity. The model also assumes that subtask boundaries can be learned from demonstrations—tasks with ambiguous segmentation (e.g., pouring liquid) might confuse the long-term memory update.
Scaling to even longer tasks (20+ steps) hasn’t been tested, and the memory vectors might saturate. The authors note that the model struggles when the language instruction changes mid-task—it cannot update its long-term plan flexibly. Finally, only open-loop execution was evaluated; closed-loop replanning with memory could behave differently.
Frequently Asked Questions
What problem does dual latent memory solve? It stops VLA models from forgetting earlier steps during long manipulation sequences, dramatically improving success rates on tasks with five or more steps.
Does this require special hardware? No. The dual memory module is a software addition—it works with any standard robot arm equipped with a camera and gripper.
How much extra computation does the memory add? Around 2.3 million extra parameters and minimal inference latency. The model runs on a single GPU at 20 Hz.
Can this work on tasks not in the training set? The model showed strong generalisation to new object colours and slightly different object poses, but hasn’t been tested on completely novel task structures.
Conclusion
Dual Latent Memory offers a simple but effective fix for a critical weakness in current VLA models: short-term blindness on long tasks. By separating immediate and long-term context, the system achieves state-of-the-art results on simulated multi-step manipulation. The next step is proving it works on real factory floors and home kitchens.