Evaluation Tasks: Measuring Different Axes of Memory
We test, in three simulated and one real task, three distinct memory-usage capabilities:
- Counting under partial observability
- Spatial recall
- In-context adaptation from past experience
We describe the tasks at a high level below and defer details to the appendix. All baselines are trained via supervised behavior cloning from demonstrations.
DrawerRecall
DrawerRecall tests spatial recall. Here, another robot first opens and puts a cube away into one of three drawers, then closes it. A separate robot—the policy under evaluation—must open the drawer containing the stored cube.
BalanceBar
Finally, we test in-context adaptation in BalanceBar. Here, a robot is given two attempts to lift a bar with an unknown center of mass (CoM) so that it remains level.
The demonstrations start with lifting from the middle first, revealing the CoM, then grasping at the CoM, resulting in a level lift. The learned policy must learn to do the same: use history to determine the CoM and reattempt at the correct grip position.
Workspace Models Successfully Amortize VLM Reasoning
The central motivation is to amortize slow and computationally intensive VLM-in-the-loop history summarization into a lightweight encoder that can be queried efficiently at deployment.
Before evaluating downstream policy performance, we verify that this amortization improves run-time.
Why Do Workspace Models Exhibit Better Task Performance?
This section accounts for the surprising finding that workspace models, despite being targeted at amortizing key-frame lookup, actually outperform key-frame lookup.
Full Histories and Key-Frame History Both Induce Control Failure
We first categorize the failure modes exhibited by different methods as either memory-related errors or control-related errors.
Memory-related errors occur when the robot correctly executes a skill but targets the wrong mode—for example, putting three blocks away and pressing done instead of putting five away.
Control-related errors occur when the robot fails to execute a manipulation skill—for example, missing a grasp of a block.
We share results of failures within simulation experiments, finding that VanillaDP unsurprisingly suffers from memory-related errors but rarely fails to execute an action mode.
Keyframe-Based Methods Introduce Control Aliasing
Explicit frame selection methods—whether strided, as in HistoryDP, or VLM-based, as in Keyframe—create an aliased input structure during training. This structure may be prone to distribution shift from the narrow training distributions.
Why Do Workspace Models Generalize Better?
The workspace model exhibits smoothness in both its own training input and its outputs, which serve as policy inputs.
This smoothness results from:
- Training with full global attention over time
- Smoothness in the workspace model’s outputs
Discussion
We present workspace models as a solution to the problem of long-horizon memory in robotic manipulation. By amortizing VLM reasoning at training time, we obtain a fast, robust encoder that generalizes better than VLM-in-the-loop alternatives.
The specific instantiation we propose—saliency sets as image patches and a DETR-style decoder—is one way to realize this idea. Many alternatives are worth exploring, as are other forms of VLM supervision.
For instance, future work could extend the method to multi-camera streams, encode language reasoning traces, or incorporate dynamics prediction.
More broadly, we hope this work encourages that perspective. The core principle extends beyond robotics and memory: anywhere a powerful foundation model is queried repeatedly at test time to solve a structured subtask, there may be an opportunity to amortize that reasoning into a faster and more robust learned module.
As we have shown, amortization can lead to even stronger results than stronger models in the loop.
While the workspace model works well, it relies on a fully autoregressive transformer encoder. This quadratically scales computation and can become cumbersome at very long sequence lengths.
To ameliorate this, we can train workspace models to be recurrent, use block-diagonal attention masks to reduce overhead, or employ KV-caching.
Workspace models also receive supervision on what visual information is salient through a VLM, which may not be fully grounded in determining which parts of an image are relevant to control. Reinforcement learning can provide an option for grounding in task reward.
Lastly, we do not compare to hierarchical methods utilizing language summaries and VLAs. It would be interesting to examine these methods, along with how workspaces can be combined with generalist policies in the multi-task setting.
Acknowledgments
We want to express our gratitude to Younghyo Park, Ryan Bahlous-Boldi, and Antonia Bronars for relevant commentary about the work, along with the Improbable AI Lab community for fruitful discussions.
This research was financially supported by the Ministry of Trade, Industry, and Energy (MOTIE), Korea, under the “Global Industrial Technology Cooperation Center program” supervised by the Korea Institute for Advancement of Technology (KIAT). Grant No. P0028435.
This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. 2141064.
Author Contributions
- Nitish Dashora: Co-developed the project direction, architecture, experimental design, hardware experiments, paper writing, and code development.
- Douglas Chen: Contributed to writing, experimental design, and code development.
- Idan Shenfeld: Co-developed the project direction, contributed to writing and experimental design, and provided conceptual guidance.
- John Marangola: Contributed to hardware experiments.
- Pulkit Agrawal: Contributed to paper writing and high-level advising.
- Max Simchowitz: Co-developed the project direction, architecture, experimental design, and contributed significantly to paper writing.
Event Detection
This section discusses the details of the VLM labeling pipeline.
The workspace token contains information about both the past and present, compressing information both temporally and spatially. Therefore, at a given time step, we distinguish between two different types of patches in the salient sets: event patches and transient patches.
Event patches are specialized to an event, are chosen at the time of the event, and remain in the salient set.
Transient patches are relevant only in the present and exist in the salient set for the current time.
Simulation Details
ManiSkill3 is configured to run at 100 Hz. All data collection in simulation is scripted with simple linear motion planners using privileged state information.
We run our policies and data collection at 20 Hz with PD control on a 7D action space including end-effector pose and gripper state.
DrawerRecall
In this task, the robot observes a secondary robot open a drawer, pick up a cube, place it into the drawer, and then close it.
This is done such that it is impossible for the primary robot to know which drawer contains the cube after the drawer is closed. The robot’s task is then to open the correct drawer.
We use 272 demonstrations for training.
BalanceBar
This task involves lifting a long bar so that it is level—for example, zero degrees of tilt about the midpoint. However, the bar has one of three centers of mass (CoM).
The demonstrations begin with an attempted pick from the midpoint, revealing a tilt. The demonstrations then include lifting the bar from the correct CoM, resulting in a balanced lift.
For the learner, success is counted if the bar is picked in a balanced way on the second try.
We use 272 training demonstrations.
Frequently Asked Questions
What capabilities do the workspace models evaluate? They evaluate counting under partial observability, spatial recall, and in-context adaptation from past experience.
What is the purpose of amortizing VLM reasoning? It converts slow, computationally intensive VLM-in-the-loop history summarization into a lightweight encoder that can be queried efficiently during deployment.
What is DrawerRecall? DrawerRecall tests whether a robot can use history to identify and open the drawer containing a cube placed there by another robot.
What limitation do workspace models have? Their fully autoregressive transformer encoder scales quadratically with sequence length, which can become cumbersome for very long sequences.