Geometric understanding from images is a foundation of computer vision. Methods such as DUSt3R, MASt3R, and VGGT demonstrate that transformers can produce rich, transferable geometric representations when trained on large-scale annotated 3D data. This supervision requirement is a fundamental bottleneck since dense 3D ground truth is expensive to collect at scale.
Novel view synthesis (NVS) offers a compelling alternative. These models must reason about 3D scene structure to predict unseen viewpoints, providing a rich geometric supervision signal from posed images, which are abundantly available across videos, robotics platforms, and structure-from-motion pipelines. However, representations produced from feed-forward NVS methods transfer poorly to downstream geometric tasks. We argue that this poor transfer is not a signal failure, but rather an architectural failure, driven by two identifiable and independently addressable choices that dilute the signal.
First, spatially expressive decoders tend to absorb the representational load of the task. When a decoder has sufficient capacity to reconstruct a target view from any encoder output, the encoder is not pressured to maintain structured, locally discriminative features. While the reconstruction task is solved, the latent representations are not geometrically grounded. Second, pixel-space reconstruction targets require modeling high-frequency aliasing statistics that are weakly related to 3D structure and semantics. Consequently, the reconstruction loss produces gradients that encourage the representation encoder to preserve fine-grained, image-specific statistics that are not directly informative about the underlying 3D structure or semantics.
SNAP is a self-supervised multi-view spatial reasoning backbone that builds upon frozen 2D semantic latent spaces with an asymmetric NVS prediction objective and latent feature targets to learn geometrically transferable representations without dense 3D supervision.
The paper presents:
- Mechanistic analyses isolating the independent impact of decoder expressivity and prediction space on geometric transfer, using receptive field sweeps and prediction space landscape experiments.
- Empirical evaluation across five tasks, matching or exceeding self-supervised baselines and competing with geometry-supervised methods that require large-scale annotated 3D data.
Related Work
SNAP sits at the intersection of self-supervised representation learning, novel view synthesis, and geometric foundation models.
Latent Reconstruction Objective
By routing supervision through DINOv3 features rather than RGB, the method replaces a weak, low-level training signal with more structured guidance. Unlike raw pixels, which contain nuisance variation and little inherent bias toward semantics, DINOv3 features reside on a structured manifold and provide dense, spatially localized embeddings. The choice is motivated and its effect quantified through a systematic sweep over DINOv3 layers.
Multitask Evaluation of Representation
The geometric content of the frozen learned representation is probed on five tasks spanning correspondence, localization, and pose estimation. In every case, the representation is held frozen; only a lightweight task-specific head is trained.
Analysis: Decoder Expressivity and Prediction Target
The objective is to learn scene representations that transfer across geometric tasks, not simply to optimize novel-view reconstruction. These goals can diverge: an encoder-decoder NVS model may reconstruct target views accurately while placing little transferable spatial structure in the multi-view encoder. Two key design factors influence whether the learned multi-view representation encodes transferable spatial structure: decoder expressivity and the prediction target.
Prediction Space Determines Representation Quality
The prediction target determines what information the model is pressured to encode. Recent NVS models typically supervise reconstruction in pixel space, which can emphasize low-level appearance, texture, and illumination rather than transferable geometry. Instead, SNAP uses latent feature prediction: earlier ViT layers remain closer to RGB, while deeper layers yield high-level structural abstractions that are robust to non-geometric artifacts such as lighting and aliasing.
The experiments sweep both the DINOv3 layer used as the featurized input and the layer used as the prediction target, asking which feature spaces are optimal for novel-view prediction at both the input and target. The resulting encoders are evaluated on RE10K and the DL3DV validation subset using feature reconstruction error, nearest-neighbor point correspondence (PCK@0.1), patch retrieval (MRR), and intrinsic dimensionality using the TwoNN estimator.
Experiments
SNAP is evaluated across five tasks using frozen encoder features with lightweight task-specific decoder heads. This protocol isolates the intrinsic spatial structure learned during pretraining, testing out-of-the-box generalization without confounding effects from task-specific fine-tuning data.
To evaluate domain transfer, the protocol is split: point correspondence and relative pose are evaluated on the validation splits of the pretraining datasets, RealEstate10K and DL3DV. Depth estimation and robot manipulation are evaluated on domains unseen by the vision backbone—Hypersim, 7Scenes, and MimicGen—to measure out-of-domain generalization.
For perceiver-style NVS methods whose latents are not spatially organized, including LVSM and RayZer, their latents are cross-attended to a shared spatial token grid. All downstream features are computed from the resulting spatial grid tokens, ensuring a fair comparison with spatially indexed encoder methods.
Visual Localization
Visual localization requires matching a query image against a reference map and estimating absolute camera pose, thereby combining per-patch discriminability with global geometric consistency under large viewpoint changes. Evaluation trains a lightweight regression head on frozen encoder features to directly output a 6D rotation matrix and 3D translation.
SNAP establishes a new frontier among self-supervised models on most metrics, confirming that the structured features induced by receptive field restriction generalize beyond local correspondence to global pose inference.
Conclusion
While current pretraining relies on posed video sequences and static, textured scenes, and the findings are bound by a fixed compute and data budget that has yet to reveal the full scaling laws of the architecture, SNAP demonstrates that feed-forward novel view synthesis is a legitimate powerhouse for self-supervised geometric pretraining. Extending this paradigm to dynamic video remains a critical next step in establishing this approach as a foundation model. More broadly, the results show that NVS’s rich geometric signal is most available when two architectural pathologies are corrected: decoders expressive enough to absorb spatial reasoning that should belong to the encoder, and pixel-space targets that contaminate gradients with low-level statistics.
Pretraining Setup
Optimization
Training was conducted for 40,000 steps using an effective batch size of 256 across multiple classes of GPU hardware using Distributed Data Parallel (DDP). The AdamW optimizer was used with a peak learning rate of the specified peak learning rate and a weight decay of 0.02. The learning rate follows a linear warmup for the first 2,000 steps, followed by a cosine annealing schedule that decays toward zero. Gradient clipping is applied with a maximum norm of 0.5 to maintain training stability.
Exponential Moving Average
To stabilize the learned representations, an Exponential Moving Average (EMA) of all trainable parameters is maintained with a decay rate of 0.999. The EMA scope encompasses every component of the architecture, including the pose-free multi-view encoder and the pose-conditioned local decoder, which serves as the latent feature reconstruction predictor. All empirical evaluations and downstream geometric probes reported in the main text use these EMA weights.
Point Correspondence
Label Generation
Training labels are generated using image pairs from the train splits of RealEstate10K, DL3DV-10K, and Hypersim. Two distinct strategies are used depending on the dataset:
- RE10K and DL3DV: Pseudo-ground-truth correspondences are derived from a frozen RoMa v2 matcher. To ensure high-quality supervision, image pairs are filtered to include at least the specified number of confident matches.
- Hypersim: Exact ground-truth correspondences are computed via cross-projection using provided depth maps and camera poses.
The decode stage consists of three cross-attention layers, and the localization stage consists of three cross-attention layers. In the final matching stage, independent linear projections are applied to the queries and keys to prevent gradient interference. The normalized query-key dot product is scaled by a learnable, log-parameterized temperature scalar, initialized to 1.0, before calculating the similarity heatmap. A differentiable spatial expectation over the resulting similarity heatmap produces the final predicted coordinate.
Optimization
The correspondence head is trained for 20,000 steps with an effective batch size of 128. The AdamW optimizer is used with a peak learning rate of the specified peak learning rate, a 2,000-step linear warmup, and a cosine annealing schedule. As specified in the evaluation protocol, all views are encoded under identity camera-to-world poses with pixel-unit intrinsics to isolate the encoder’s intrinsic geometric awareness from external pose signals. Evaluation occurs on the held-out validation splits of all datasets.
Visual Localization
Architecture
The visual localization head is designed to regress an absolute target pose by localizing an unposed query image against a reference multi-view context. The target pose is parameterized using nine parameters: a continuous 6D rotation representation and a 3D translation.
The regression head uses alternating transformer layers of self-attention, which process the target sequence, and cross-attention, which reads from the reference context. A learned CLS token is prepended to the target image sequence, and 1D RoPE is applied to encode sequence positions. After the tokens are processed through the transformer blocks, the refined CLS token is extracted and passed through a final multilayer perceptron to output the nine pose parameters. To ensure stable early training, the final linear layer’s biases are initialized to represent an identity transformation.
Pose Estimation
Optimization
The sequence head is trained on the RealEstate10K and DL3DV train splits and evaluated on their held-out validation splits. Training uses a peak learning rate of the specified peak learning rate, governed by a cosine annealing schedule with 2,500 warmup steps. The model is optimized using a 9D pose regression loss based on the Smooth-L1 error of the rotation and translation predictions.
Frequently Asked Questions
What is SNAP designed to learn? SNAP learns geometrically transferable representations from posed images using self-supervised multi-view spatial reasoning, without dense 3D supervision.
Why does SNAP use latent feature prediction instead of RGB reconstruction? DINOv3 features provide structured, spatially localized guidance, while pixel-space targets can emphasize low-level appearance, texture, illumination, and aliasing.
How does decoder expressivity affect geometric transfer? A decoder with enough capacity can absorb the reconstruction task, reducing pressure on the encoder to maintain structured, locally discriminative features.
Which downstream tasks are used to evaluate the representations? The paper evaluates five tasks spanning correspondence, localization, and pose estimation, as well as depth estimation and robot manipulation.