Patch Policy investigates the efficacy of patch-level visual features for robot learning, introducing an efficient policy class that harnesses pre-trained dense visual representations to achieve superior performance while remaining computationally lean.
Policy Learning
This formulation is agnostic to the action head architecture and training objective. In our experiments, we evaluate Patch Policy using two state-of-the-art architectures: Vector-Quantized Behavior Transformer (VQ-BeT), which uses a hybrid classification-regression loss, and Diffusion Policy, which uses a denoising objective.
During training, we forward a sequence of patch tokens through the policy transformer trunk and action head, and compute a loss between the predicted and ground-truth actions for each frame. For inference, we extract the patch features from the current observation and append them to a rolling context window of length a configurable sequence length. The policy predicts a chunk of actions from the observation context, and we execute the actions with receding horizon control.
Experiments
We evaluate Patch Policy across four simulated environments and three real robot manipulation environments, aiming to answer the following key research questions:
- How does Patch Policy compare to state-of-the-art policies using global and patch features?
- Does Patch Policy work for real-world precise manipulation?
- How do the latest pretrained encoders perform as representations for downstream policy learning?
- How does the spatial compression of visual features impact downstream task performance?
- How does Patch Policy compare to other methods in terms of efficiency? How do attention design and model size affect its performance?
Environments
We evaluate Patch Policy across four simulated environments (Push-T, LIBERO Goal, BlockPush, Cube) with 2D-to-7D action spaces, and three real-world tasks using a 7-DoF Franka arm with a parallel-jaw gripper (inserting a power cable, hanging a tool, and collecting pens into a holder).
Baselines
We compare against two groups of baselines. To isolate the effect of patch features, we pair the same policy heads (VQ-BeT, Diffusion Policy) with standard global representations: DynaMo, CLS tokens, and global average pooling. To position Patch Policy against the state of the art, we compare to ACT and OpenVLA-OFT, which also consume dense features. Our baseline representations and policies are as follows:
Visual Representation Baselines: - DynaMo: A global pooled representation learned via dynamics-based joint-embedding predictive architecture. - CLS Tokens: The class token from the Vision Transformer, representing a compressed summary of the scene. - Average Pooling: A baseline that collapses the spatial feature map into a single vector via global average pooling. - ACT: A conditional VAE that predicts action chunks with temporal ensembling over overlapping chunks, reducing compounding error in fine manipulation. It uses patch features from a ResNet-18 vision encoder trained from scratch. - OpenVLA-OFT: A Vision-Language-Action model that finetunes OpenVLA with parallel action decoding and L1 action regression for faster training and inference. It builds on a Llama-2 7B LLM backbone and consumes channel-fused patch features from pretrained DINOv2 and SigLIP vision encoders.
Benchmarking Pre-Trained Visual Representations
For all experiments, we freeze the image encoder and exclusively train the policy. This approach isolates and evaluates the out-of-the-box representation capabilities of each backbone. This frozen setup also allows us to precompute the visual embeddings, significantly accelerating training.
Notably, the relative ranking of visual representations remains remarkably consistent across different policy architectures for any given task. This consistency suggests that the quality of the visual representation is still a primary bottleneck for policy learning, independent of the downstream action head. Furthermore, the average ranking of these representations remains stable across the entire task suite, underscoring the generalizability of certain pre-training objectives. Based on these findings, we recommend the use of WebSSL or DINOv2 as the vision backbones for robot learning tasks.
Additional Ablations
The appendix contains more experiments and details on attention masks, model size, and visualizations. Transformer-based policies traditionally condition on a single token per observation step, allowing direct application of standard token-level causal attention. In Patch Policy, each observation is now a sequence of tokens and we thus apply a block-causal attention mask. Under this setting, every patch token attends fully to all other patch tokens within the same frame, while attention across frames remains strictly causal. We ablate this block-causal attention along with two other masking choices: full attention with no masking, and token-causal, the naive implementation that treats every patch as an independent timestep.
Limitations
While Patch Policy effectively leverages dense spatial features for control, several directions remain for future work. First, we focused exclusively on frozen vision backbones, and future work could explore end-to-end fine-tuning to adapt these representations for specialized visual domains. Second, dense tokens increase sequence length and training time. Optimizations like FlashAttention could accelerate both training and inference. Finally, Patch Policy is currently evaluated purely as a behavior cloning policy. Extending this patch-based architecture to reinforcement learning could be a promising direction to surpass the performance ceiling of static expert demonstrations.
Conclusion
In this work, we investigate the efficacy of patch-level features for robot learning and introduce Patch Policy, an efficient policy class that harnesses pre-trained dense visual representations. It achieves superior performance while remaining computationally lean across parameter count, training compute, and inference latency, outperforming both global-feature policies and heavyweight VLAs. By keeping the visual backbone frozen and ingesting its patch tokens directly through a block-causal attention mask, Patch Policy provides the robotics community with a lightweight, drop-in pipeline to readily absorb continuing progress in visual representation learning, without sacrificing the fast training and high-frequency control that real-world manipulation demands.
We would like to thank Mahi Shafiullah, Irmak Guzey, Kevin Wu, Hengkai Pan, Zifan Zhao, Alex Xiaole Jiang, Andre Wang, Zicheng Teng, Kanad Patel, Yvonne Wu, Zavier Andrianarivo for their valuable discussion and feedback. This work was supported by grants from LG, Qualcomm, Honda, Microsoft, Hyundai, NSF award 2339096 and ONR awards N00014-21-1-2758 and N00014-22-1-2773, and AFOSR under grant FA95502310139. Lerrel Pinto is supported by the Sloan, Packard, and CIFAR Fellowships.
Frequently Asked Questions
How does Patch Policy differ from using CLS tokens or global pooling? Patch Policy preserves the full spatial structure of visual features by processing all patch tokens with block-causal attention, rather than compressing them into a single global vector. This dense representation provides richer spatial information for precise manipulation tasks.
What vision backbones perform best with Patch Policy? WebSSL and DINOv2 rank as the top-performing frozen backbones across all evaluated tasks, and the ranking remains consistent regardless of which policy head is used. The quality of visual representation is a primary bottleneck for policy learning.
Does Patch Policy work on real robots, not just simulation? Yes, Patch Policy was validated on three real-world precise manipulation tasks using a 7-DoF Franka arm: cable insertion, tool hanging, and pen collection into a holder. It achieved successful rollouts across all three tasks.
Why does Patch Policy freeze the vision encoder instead of fine-tuning it? Freezing the encoder isolates the evaluation of each backbone's representation quality and allows precomputing visual embeddings, which significantly accelerates training. Fine-tuning remains an option for specialized visual domains.