We present VGGT-Edit, a feed-forward framework for instruction-driven native 3D scene editing that treats editing as a 3D residual field prediction task, avoiding multi-view inconsistencies common in 2D-lifting pipelines while preserving structural integrity.
3D Editing Data Pipeline
To train VGGT-Edit, we construct an automated 3D editing data generation pipeline that produces large-scale pairs of original and edited 3D scenes. Given raw multi-view observations, the pipeline converts them into high-quality, instruction-aligned, and view-consistent 3D editing pairs through four key stages.
3D Mask Refinement
This refinement step reduces noisy supervision and enforces stronger multi-view consistency.
Sequential Multi-View Editing
A central challenge in data generation is maintaining appearance and geometry consistency of the edited target across viewpoints. Editing each view independently can introduce inconsistent colors, textures, shapes, or spatial layouts, making the resulting data unsuitable for learning native 3D scene editing. To address this issue, we adopt a sequential multi-view editing strategy. Instead of editing all views independently, we edit them in an ordered sequence and condition the current edit on the previously edited view.
By propagating visual context across adjacent views, this strategy encourages consistent object appearance and spatial placement throughout the sequence. As a result, the generated editing pairs provide more reliable supervision for learning residual field prediction in 3D space.
Viewpoint Selection and Quality Control
Not all views provide equally reliable supervision. Some viewpoints may contain severe occlusion, truncation, extreme viewing angles, or weak target visibility. To select high-quality observations, we introduce a Re-projection Fidelity score to evaluate each view. For a given view, we project its mask into 3D and then re-project it back to the image plane, obtaining a reconstructed mask. The score is defined as a function of the viewing angle.
This metric favors views with accurate geometric projection and frontal, unobstructed observations. By filtering out unreliable views, the pipeline provides cleaner supervision and improves the stability of VGGT-Edit training.
The DeltaScene Dataset
In this section, we provide a detailed description of the DeltaScene Dataset, which is specifically constructed to address the lack of large-scale, view-consistent data for native 3D scene editing. High-quality data is fundamental to training our residual learning paradigm, as it requires precise geometric alignment between the original and edited scenes.
Architecture Overview
VGGT-Edit consists of three main architectural components. First, a frozen feed-forward reconstruction backbone provides a strong geometric prior. Second, a depth-synchronized text injection module aligns the editing instruction with spatially grounded multi-view features. Third, a residual transformation head predicts a dense residual displacement field, which is added to the base geometry under the guidance of an edit mask. To train this architecture, we further introduce a residual-oriented objective that combines edit reconstruction, non-edit preservation, normal consistency, camera-frame consistency, and residual regularization. This formulation enables VGGT-Edit to preserve unchanged regions and perform localized geometry deformation.
Depth-Synchronized Text Injection
To perform instruction-driven editing, VGGT-Edit must map the semantic intent of a text instruction to the correct spatial region in the 3D field. We therefore introduce a depth-synchronized text injection module, which injects textual guidance into the reconstruction features at layers aligned with the backbone's pose-modulation stages.
Given an instruction, we obtain a text embedding using a pre-trained OpenCLIP encoder. Instead of injecting this embedding only once, we fuse it into the transformer decoder at multiple synchronized layers. These layers are selected to match the major pose-injection blocks of the reconstruction backbone. As a result, semantic guidance is introduced at the same feature depths where spatial geometry is progressively formed.
At each selected layer, we perform text-driven cross-attention between the multi-view features and the instruction embedding. This synchronized design provides continuous semantic guidance throughout the decoding process. Compared with a single early injection, it reduces the risk that textual information fades in deeper layers. Compared with injecting text into every layer, it avoids unnecessary computation and training instability. In practice, this enables the model to produce edits that are both semantically aligned with the instruction and spatially consistent across views.
View-Aware Importance Weighting
Multi-view observations are not equally informative for editing. In some views, the target object may be clearly visible, while in others it may be occluded, truncated, or close to the image boundary. Treating all views equally can therefore introduce noisy semantic guidance.
To address this issue, we introduce a view-aware importance weighting mechanism. For each view, we construct a geometric descriptor that includes the visible mask area, the boundary ratio, and the backbone confidence score. These quantities jointly describe the reliability of the target observation in that view. A lightweight MLP predicts a normalized importance weight.
The resulting weight is used to modulate the key and value features derived from the text embedding. This formulation allows views with more complete observations to contribute more strongly to the editing process, while suppressing unreliable views caused by occlusion or boundary artifacts.
Residual Field Prediction
After text injection and view-aware weighting, the spatially fused features are passed to a residual transformation head. Unlike the original reconstruction head, which predicts the full scene geometry, our head predicts only the incremental geometric change required by the instruction.
Specifically, the head outputs a dense residual displacement field. Given an edit mask, the edited point map is obtained by element-wise multiplication of the residual displacement with the mask, added to the base geometry. This residual formulation is central to VGGT-Edit. Since most regions in a scene remain unchanged after an edit, directly predicting the complete edited geometry is unnecessary and may destabilize training. By predicting only localized residual displacements, VGGT-Edit preserves the background structure inherited from the frozen backbone and concentrates its modeling capacity on the edited region.
Training Objectives
We train VGGT-Edit with a multi-term objective that supervises edited geometry, preserves unchanged regions, and enforces geometric and projective consistency. Feed-forward reconstruction models may exhibit global scale ambiguity relative to ground-truth geometry. Since our goal is to learn accurate relative editing, we compute a per-sample masked least-squares scale factor within the edit region.
The aligned prediction is then used in a masked loss function that computes the error between predicted and ground-truth edited geometry.
Experiments
In this section, we evaluate VGGT-Edit on the proposed DeltaScene Dataset. We conduct experiments to assess its performance in geometric accuracy, multi-view consistency, semantic alignment, and inference efficiency.
Main Results
We evaluate VGGT-Edit from three aspects: semantic alignment, multi-view consistency, and inference efficiency. For semantic alignment, we report the CLIP Score between rendered views of the edited scene and the input text instruction. To measure cross-view visual and geometric consistency, we compute C-FID and C-KID across rendered viewpoints. These metrics capture view-dependent artifacts such as ghosting, flickering, and inconsistent object structure, which are common in 2D-lifting pipelines. Finally, we report the average inference time per edit to measure efficiency against optimization-based and diffusion-based methods.
Qualitative Analysis
To further illustrate the impact of each component in VGGT-Edit, we provide qualitative comparisons. The visual evidence aligns with our quantitative findings:
- Without depth-synchronized attention, the model exhibits incomplete editing or color mismatches, failing to fully manifest the requested changes (e.g., the added object appears faded or incorrectly textured).
- Without view-aware weighting, the model leads to noticeable geometric artifacts and floaters, particularly at the edges of occluded regions where the model fails to resolve spatial ambiguities.
- Replacing our residual paradigm with a full reconstruction head causes significant background drifting. Background objects that should remain static exhibit subtle deformations or shifts, confirming the importance of our 3D residual learning for maintaining structural integrity.
Conclusion
We present VGGT-Edit, a feed-forward framework for instruction-driven native 3D scene editing. By treating editing as a 3D residual field prediction task, our model avoids the multi-view inconsistencies common in 2D-lifting pipelines while preserving the structural integrity of the original scene. Through depth-synchronized text injection and view-aware weighting, VGGT-Edit achieves precise semantic-to-spatial alignment and robust feature fusion, enabling localized geometry deformations in a single forward pass. Experimental results on the DeltaScene Dataset show that our method outperforms existing optimization-based and feed-forward baselines in terms of geometric fidelity, multi-view consistency, and inference efficiency. With its ability to handle diverse operations and maintain high-speed performance, VGGT-Edit provides a practical and generalizable solution for real-time, interactive 3D scene editing.
Frequently Asked Questions
What makes VGGT-Edit different from 2D-lifting approaches for 3D scene editing? VGGT-Edit operates directly in 3D space by predicting residual displacement fields, avoiding the multi-view inconsistencies like ghosting and flickering that commonly arise from editing each 2D view independently.
How does the depth-synchronized text injection module work? It injects text instruction embeddings into the transformer decoder at multiple layers synchronized with the backbone's pose-modulation stages, providing continuous semantic guidance throughout the geometry formation process.
What type of training data does VGGT-Edit require? The model uses the DeltaScene Dataset, a large-scale collection of paired original and edited 3D scenes generated through an automated pipeline that ensures view consistency, mask refinement, and quality control.
Can VGGT-Edit preserve unchanged background regions during editing? Yes, because it uses a residual formulation that predicts only localized geometric changes for the edit region, while the background structure is preserved through the frozen reconstruction backbone.