4.4 What Enables Generalist Policy Improvement?
Having evaluated our recipes at scale, we now validate key design decisions for multitask VLA post-training and investigate several alternative techniques for critic-guided policy extraction. See Appendix C for additional ablations, including data and compute scaling and resource allocation.
5 Discussion
The main goal of this work is to contribute to the understanding of mechanisms and trends of online self-improvement in dynamical systems. We believe that our results represent a promising starting point for future work, which can build on the simple and scalable RL recipes we derived and study the benefits of multitask post-training at scale on real hardware.
Reproducibility Statement
We ensure reproducibility by releasing code for our full training pipeline at the URL listed at the end of Section 3.3. All data used is collected online, and no offline dataset is necessary to reproduce the results. We also report training details, hyperparameters, and task splits in Appendix D.
D.4 Evaluation Protocol
On both LIBERO and MolmoSpaces, we evaluate each policy in-distribution and on unseen tasks. For LIBERO, the unseen set comprises 25 tasks. For MolmoSpaces, it comprises eight unseen scenes of objects from the 144-task split and six unseen objects (see Appendix D.1). The metric throughout is success rate.
During training, in-distribution success rate is measured on the 20 episodes the policy collects per task in each round, so no additional mid-evaluation rollouts are needed. After the last training phase, we run a separate evaluation phase of 32 episodes per task, both in-distribution and on unseen tasks. We repeat every experiment with three seeds and report the mean success rate over seeds, together with 95% stratified bootstrap confidence intervals computed with rliable.
D.5 Computational Resources
Our experiments ran on nodes with four NVIDIA GH200 GPUs each. A single-task run took about 2–5 hours of wall-clock time, depending on the task. Runs on the largest multitask splits took 200 hours.
Frequently Asked Questions
How was policy performance measured? The metric was success rate, measured both in-distribution and on unseen tasks.
How many evaluation episodes were run after training? The separate evaluation phase used 32 episodes per task for both in-distribution and unseen tasks.
How many random seeds were used? Every experiment was repeated with three seeds, and mean success rate and 95% stratified bootstrap confidence intervals were reported.
What computational resources were used? Experiments ran on nodes with four NVIDIA GH200 GPUs each. Single-task runs took about 2–5 hours, while the largest multitask runs took 200 hours.