We present COMPACT-VA, a planning-aligned working memory framework for monolithic autonomous driving policies. By coupling compression with trajectory prediction through conditional VQ-VAE, we address the fundamental limitation that unified VA/VLA policies either lack explicit memory mechanisms or rely on rule-based compression unable to guarantee retention of decision-relevant history. Through systematic evaluation on memory-critical stop-controlled scenarios, we demonstrate that planning-aligned compression achieves sizable improvements in decision correctness while maintaining real-time token budgets.
Unified Vision-Action Model Backbone
Our approach builds upon the unified vision-action (VA) policy variant of Alpamayo, which consists of three core components: a vision encoder processing multi-camera observations into visual tokens, a transformer backbone performing temporal reasoning without text instruction, and a trajectory decoder generating future vehicle motion.
While this unified architecture eliminates explicit intermediate modules, the sequence length grows linearly with the context length. For complex driving scenarios requiring extended temporal context, the token count grows substantially, easily surpassing typical VLM context windows. Without effective compression, the quadratic transformer attention cost O(T²d) becomes intractable for deployment.
Memory-Dependent Driving Scenarios and Evaluation Framework
We focus on scenarios where extended historical context plays a particularly important role in determining correct behavior. Prior studies identify that critical driving decisions rely on behavioral cues captured within a 5-10 second temporal window, constituting extended context relative to standard driving policies that typically process only 1-2 seconds, and distinct from navigation tasks requiring long-term spatial memory over entire routes. We identify high-signal dynamic scenarios where extended context determines behavioral correctness and design metrics for these decision-critical outcomes beyond trajectory displacement.
Stop-Controlled Intersections as Memory Testbeds
We identify three scenario classes where extended historical context plays a prominent role: (1) four-way stops requiring right-of-way negotiation based on arrival order, (2) stop/yield signs requiring assessment of dynamic cross-traffic, and (3) unprotected turns requiring gap acceptance decisions. These share a fundamental characteristic that correct behavior depends on discrete decision correctness, i.e., whether the vehicle stops when required and proceeds when appropriate, instead of trajectory smoothness.
Four-Way Stop Intersections. Multiple vehicles arrive at an intersection with all-way signs on all approaches. Right-of-way follows arrival order, demanding tracking which vehicles arrived earlier over several seconds. The ego vehicle needs to determine its position in this temporal queue. The model must maintain information about arrival times, 5-10 seconds ago, to correctly infer yielding order and proceed promptly when gaining right-of-way.
Stop/Yield with Dynamic Occlusion. As ego vehicle approaches the intersection, previously visible participants may become occluded or exit the field of view. The model must persist their state observed seconds earlier rather than relying solely on currently visible objects to avoid incorrectly assessing an occluded intersection as clear.
Unprotected Turns. The ego vehicle turns across oncoming traffic without a protected signal. An oncoming vehicle observed seconds ago may begin decelerating to yield, or maintain speed requiring the ego vehicle to stop and wait for a safe gap. Without tracking oncoming vehicles' trajectories over several seconds, the model risks either turning into oncoming traffic or waiting unnecessarily.
These three scenario classes encompass the core decision-making challenges in driving. According to prior analysis, fundamental skills tested here, i.e., errors in gap acceptance, right-of-way negotiation, and stopping behavior, account for approximately 40% of intersection crashes.
Overall Performances
We prioritize Go Success Rate as the primary metric, as it directly tests whether the model maintains effective memory to determine when to proceed, requiring long-horizon memory to assess cross-traffic patterns and right-of-way. In contrast, Stop Success Rate, while important, can often be achieved reactively by observing immediate deceleration trends without extended memory.
The standard Alpamayo already achieves competitive performance (63.8% Go SR), demonstrating the effectiveness of its architecture. Sparse observation with long history (62.0%) underperforms, revealing that overly sparse sampling discards critical intermediate frames and actually hurts performance. Surprisingly, dense observation with full history (61.9%) performs worst despite accessing all 40 frames, suggesting that indiscriminate token abundance confuses the model, and quadratic attention cost over 6400 tokens hinders temporal reasoning. Learned hierarchical compression without planning alignment (65.6%) improves over Alpamayo, validating that structured compression retains decision-relevant information.
Conclusion
This work presents COMPACT-VA, a planning-aligned working memory framework for monolithic autonomous driving policies. By coupling compression with trajectory prediction through conditional VQ-VAE, we address the fundamental limitation that unified VA/VLA policies either lack explicit memory mechanisms or rely on rule-based compression unable to guarantee retention of decision-relevant history. Through systematic evaluation on memory-critical stop-controlled scenarios, we demonstrate that planning-aligned compression achieves sizable improvements in decision correctness while maintaining real-time token budgets. Our working memory approach with bounded context windows is well-aligned with driving, where critical decisions typically depend on behavioral cues within 5-10 seconds. We believe this work advances the field toward effective memory for VA/VLA policies, demonstrating task-aware working memory is key to reasoning in diverse scenarios.
Future Work: While gains on general driving scenarios are modest, future work could explore higher-complexity scenarios with severe occlusions or multiple contenders where preliminary observations suggest larger gains, as well as recurrent memory mechanisms and alternative efficient architectures such as state-space models to extend planning-aligned compression to broader embodied AI domains.
Frequently Asked Questions
What problem does COMPACT-VA address? COMPACT-VA solves the problem of unified autonomous driving policies that either lack explicit memory or use rule-based compression that discards decision-relevant history from extended temporal context.
How does planning-aligned compression work? It couples token compression with trajectory prediction using a conditional VQ-VAE, ensuring that compressed tokens retain information most relevant for future driving decisions rather than just minimizing reconstruction error.
What driving scenarios most benefit from this approach? Stop-controlled intersections — including four-way stops, stop/yield with occlusion, and unprotected turns — show significant gains because correct behavior depends on recalling events 5-10 seconds in the past.
How does COMPACT-VA compare to dense observation baselines? COMPACT-VA outperforms both sparse sampling and dense full-history baselines, achieving higher Go Success Rate (the primary decision-correctness metric) while maintaining a bounded token budget suitable for real-time deployment.