VaM (Value as Memory) improves general robot policies by feeding recent estimates of task progress back into the policy as a compact memory, helping it choose actions when the current view is ambiguous. Across three real-world manipulation tasks, the researchers report stronger post-training performance and more sample-efficient learning, while combining demonstrations, past robot experience, and new deployment rollouts.
What Did the Researchers Build?
VaM is an online post-training system for general robot policies. Post-training means continuing to improve a policy after it has first learned from demonstrations. The system adds a value expert alongside the action expert: the action expert selects what the robot should do, while the value expert estimates how much progress the robot is making toward completing its task.
The key idea is to cache recent value predictions and feed them back to the action expert. That cache acts as a short summary of task progress, rather than a record of every image and action the robot has experienced. It can help when a robot’s latest camera view does not clearly show what has happened so far—for example, during a multi-step manipulation task where an earlier action changes the state of an object.
VaM was tested on three real-world tasks: box assembly, laundry folding, and magnetic bus assembly. They cover long sequences of actions, including both contact-heavy assembly and deformable objects. The box task, for instance, requires the robot to raise a cardboard box, fold and clamp its side flaps, then lay the box flat. Its success criterion includes finishing within 120 seconds, with the box horizontal and both flaps locked in place.
What Results Did VaM Deliver?
The researchers report that value caching improves policy performance during reinforcement-learning post-training. Their real-world ablations also support the idea that recent value predictions help address action ambiguity when the robot only has a partial view of task progress. In practical terms, the policy can use a compact signal about how the task is going, rather than relying only on what is immediately visible.
The work also reports improved sample efficiency from VaM’s offline-to-online training system. Sample efficiency matters because each physical robot trial takes time: better use of demonstrations and previous experience can reduce how much new interaction is needed to improve a policy. The system is also designed to increase policy iteration frequency by training from multiple sources of experience.
The evaluation spans three tasks with different manipulation demands. For the box-assembly evaluation, the robot faces four tablecloths with different textures and appearances, testing the task across more than one visual and surface setup. However, the provided paper text does not include success-rate percentages, numerical gains over comparison methods, or detailed benchmark tables. It therefore supports a qualitative account of the reported improvements, but not a precise estimate of their size.
How Does Value-Cached Post-Training Work?
VaM uses two cooperating models. The action expert produces the robot’s actions. The value expert estimates future task return—a measure of how much reward or progress a state is expected to lead to. Rather than expressing that estimate as only one number, the value expert is distributional: it learns a distribution over possible returns. The recent value predictions are stored in a cache and supplied to the action expert as context.
That context is intended to help resolve partial observability. A robot’s current camera image might not make it obvious whether an earlier grasp succeeded or whether a multi-step operation is nearly finished. A recent value estimate gives the policy another clue about the task’s direction. It is not a guarantee that the estimate is correct: the cache is only useful when the value expert tracks real progress well.
Training proceeds through four stages:
- Offline demonstration initialization: the policy begins with demonstration data, giving it a starting point before online reinforcement learning.
- Offline RL warmup: the system collects and trains on offline data, first adapting the value expert and then the action expert. This step helps establish a useful progress signal before it is used during online learning.
- Online DAgger: the training loop incorporates experience gathered during online interaction.
- Learn from Experience (LfE): the system continues learning from collected experience as it adapts the policy.
During online reinforcement learning, the value and action experts are updated in parallel. VaM reuses offline demonstrations, off-policy trajectories (data from earlier interactions), and current on-policy rollouts (data from the policy currently being run). Advantage labels also guide policy post-training: they provide a learning signal about whether actions performed better or worse than the value estimate would predict. Together, the cache and training loop connect task-progress estimates to policy updates while making use of both older and newly collected data.
Why Does This Matter for Real-World Robotics?
A general robot policy has to manage more than choosing a plausible next movement. It also has to keep track of where it is in a task, especially when the camera view is incomplete or the task involves several steps. VaM addresses that problem with a learned progress signal that can travel with the policy’s recent context. This is a practical alternative to relying only on the latest observation to infer what has already happened.
The training setup also reflects a central deployment challenge: real-world interaction is costly, but a robot can accumulate useful experience over time. Reusing demonstrations and past trajectories alongside fresh rollouts gives learning systems more ways to benefit from collected data. That could matter for teams developing robots for repeated manipulation workflows, where each deployment generates new examples of successful and unsuccessful behavior.
VaM is a learning method, not a particular robot platform. Buyers comparing manipulation-capable hardware can browse used robots for sale on Robot Overflow while considering whether a platform can collect the observations and interaction data that an online learning system needs. The experiments show progress on a single robot, however, so the findings do not yet establish how shared learning would work across a fleet.
What Are VaM’s Limitations and Open Questions?
VaM relies on a value function that accurately tracks task progress. If its predictions are poor, the cached signal can mislead the action expert instead of helping it. The researchers identify offline RL warmup as necessary for stable online training, which means the system depends on a useful preparatory learning stage rather than beginning effective online adaptation immediately.
The reported experiments use a single robot. They do not show whether VaM scales when several robots contribute experience, how data from different robots should be combined, or whether the method remains reliable across different hardware. Fleet learning is identified as a promising direction, not a demonstrated capability.
The available paper text also omits numerical success rates and detailed comparisons, making it difficult to judge the size of the reported gains or how they vary by task. Further evaluation would need to show both quantitative improvements and how often cached value predictions are wrong in real deployments.
What Do Buyers and Engineers Commonly Ask?
Does VaM replace demonstrations? No. VaM starts from offline demonstrations and then continues learning with offline and online experience.
What does the value cache remember? It stores recent predictions from the value expert about task progress, rather than a full history of images and actions.
Which tasks were tested? The evaluation covered box assembly, laundry folding, and magnetic bus assembly on a real robot.
Does the paper report exact success-rate improvements? The supplied text reports qualitative improvement but does not provide numerical success rates or effect sizes.
What Is the Practical Takeaway?
VaM gives a general robot policy a compact, learned signal about recent task progress and uses it to guide online post-training. The approach connects that memory to a broader learning pipeline that reuses demonstrations, previous trajectories, and fresh robot experience, but its value depends on reliable predictions and remains tested on a single robot.