We introduced Geometric Action Model, which unifies geometry and action prediction with temporal world modeling inside a single shared GFM. By inserting a causal transformer between the GFM's shallow and deep layers, GAM autoregressively decodes actions and future geometries, resolving the spatial ambiguities of traditional foundation-model substrates. Across extensive simulation and real-world benchmarks, GAM achieves superior accuracy, faster inference, and strong out-of-distribution robustness to environmental perturbations.
Main Results
Real-world Results. To examine whether the gains observed in simulation transfer to physical execution, we additionally evaluate GAM in a real-world setting. GAM substantially outperforms all baselines. In particular, our model remains robust under out-of-domain conditions (the camera-perturbation setting) where other baselines struggle. These results demonstrate that GAM generalizes to the real-world domain and is robust under perturbations, owing to its thorough exploitation of the GFM when training the policy.
Inference Speed and Model Size
GAM achieves the lowest latency among all baselines, requiring only 6.9 ms (approximately 145 Hz) for a single feed-forward pass and running up to 55 times faster than the diffusion-based Cosmos Policy. All methods are benchmarked under the same setup. By utilizing single-pass prediction, GAM avoids the multi-step denoising of diffusion policies, achieving low latency while matching prior accuracy and robustness with only 1.4B parameters.
Robustness to Viewpoint and Scene Variation
GAM achieves consistently higher success rates than all baselines at every level of camera perturbation in the LIBERO-Plus benchmark, and the advantage remains clear even under the strongest perturbations.
Simulation Experiments Details
We adopt the LIBERO evaluation protocol established by OpenVLA and OpenVLA-OFT. Specifically, we evaluate on the four standard LIBERO task suites: LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-Long, each consisting of 10 tasks. Following OpenVLA-OFT, we train on filtered LIBERO demonstrations by removing unsuccessful episodes and filtering idle/no-op frames. We fine-tune a separate policy for each LIBERO suite.
We report task execution success rate (SR, %) as our primary evaluation metric. For the original LIBERO benchmark, each task is evaluated over 50 randomized trials, resulting in 500 rollouts per suite. For LIBERO-Plus, we follow the official evaluation setting and use one rollout per perturbed task instance. All models are trained with a global batch size of 160 for up to 110k training steps, until convergence is achieved.
Model Size Breakdown
Of the full model, approximately 983.2M parameters are trainable. Most of these trainable parameters come from the later blocks of the ViT-Giant backbone, while the initial geometric layers and the DPT depth head remain frozen to preserve pretrained geometric structure. The Causal Future Predictor and lightweight action head are fully trainable and account for the remaining trainable parameters.
Generated Future Depth Maps
Given a current RGB observation, GAM predicts the future depth maps while simultaneously generating actions that align spatially with the anticipated future geometry. As demonstrated in the visualizations, GAM accurately forecasts the future depth alongside its corresponding action sequence.
When to Predict Actions?
We additionally evaluate a direct-action supervision variant that applies the action loss directly to the output action token of the causal future predictor, without passing the action token through the remaining deep geometric decoder blocks. Passing the action token through the deep geometric decoder provides an additional improvement, particularly on LIBERO-Plus Object, suggesting that the remaining GFM layers contribute to refine the action representation, especially under camera perturbations.
Attention Analysis
Several intermediate layers attend to task-relevant regions, with clear saliency around manipulated objects and nearby contact regions. This qualitative trend is consistent with the layer ablation: mid-level representations retain object-level structure while still leaving enough depth in the GFM decoder for action-token refinement.
Conclusion and Limitation
We introduced Geometric Action Model, which unifies geometry and action prediction with temporal world modeling inside a single shared GFM. By inserting a causal transformer between the GFM's shallow and deep layers, GAM autoregressively decodes actions and future geometries, resolving the spatial ambiguities of traditional foundation-model substrates. The framework also has limitations. Its language reasoning and commonsense capabilities are bounded by the frozen text encoder; integrating a large language model or an external reasoning module is a natural next step.
Frequently Asked Questions
How does GAM achieve faster inference than diffusion-based policies? GAM uses single-pass prediction to directly output actions, avoiding the multi-step denoising process required by diffusion policies. This enables 55x faster inference while maintaining comparable accuracy.
What makes GAM robust to camera perturbations and scene variations? GAM exploits the geometric foundation model's structured spatial representations and refines action tokens through the deep geometric decoder under camera perturbations. This allows it to consistently outperform baselines even under the strongest perturbations.
What percentage of GAM's parameters are trainable for policy learning? Approximately 983.2M parameters out of the 1.4B total are trainable. The initial geometric layers and depth head remain frozen to preserve pretrained geometric structure.
What is the main limitation of the Geometric Action Model? GAM's language reasoning and commonsense capabilities are bounded by its frozen text encoder. Integrating a large language model or external reasoning module would address this limitation.