Researchers introduce NEO, a method that builds a NeRF representation of a scene once and then allows multiple object edits—removal, insertion, and repositioning—without retraining. This enables robots to plan and visualize manipulation sequences from a single initial scan, drastically reducing data and computation requirements for dynamic environments.
What the Researchers Built
NEO (NeRF It Once, Edit It Many Times) is a scene-editing framework that lets robots interact with a single Neural Radiance Field (NeRF) model through multiple sequential edits. Instead of retraining the entire network after every object removal or repositioning, NEO locally updates the density and color fields based on user-provided input—either via language prompts or relative 6-DoF poses. The edited representation retains the same structure as the original, so the robot can immediately use it for the next manipulation step.
The team also built the NEO-Dataset, a benchmark of registered tabletop scenes captured before and after real object manipulations using fixed robotic arms. It includes both object removal and object movement scenarios with ground-truth geometry and poses. Objects come from the YCB dataset, ensuring reproducibility. NEO is designed to serve as a live state estimator for multi-step assembly tasks, allowing a robot to plan grasps, generate bounding boxes, and execute actions directly from the edited NeRF without re-scanning the workspace.
Key Results
On the NEO-Dataset object removal task, NEO significantly outperformed prior methods like DFF and NeRFiller in reconstructing occluded regions. For object manipulation, NEO achieved high PSNR and SSIM scores in both the “Out” region (the object’s original location) and the “In” region (the object’s new position), demonstrating its ability to render newly revealed surfaces and relocated objects faithfully.
In a four-step sequential assembly experiment, NEO predicted each intermediate scene state with minimal visual difference from real captures. The synthesized views maintained geometric consistency across all steps, even as objects were moved multiple times. Compared to DFF, which required retraining after each edit and introduced artifacts, NEO delivered stable results without additional training. The model also supported grasp planning directly on the edited representation, enabling fully autonomous robot execution after user confirmation.
How It Works
NEO builds on a standard NeRF that encodes a scene as a continuous volumetric function mapping 3D coordinates to density and color. The key innovation is a lightweight editing mechanism that updates only the relevant regions of this function. For object removal, NEO suppresses densities inside a user-specified 3D bounding box and then uses a pre-trained diffusion model to inpaint the missing colors in a multiview-consistent manner. For object movement, densities are transferred from the original location to the target pose, and the vacated region is inpainted.
A language field is attached to the NeRF, enabling object selection via text prompts (e.g., “the red mug”). This field is refined during each edit to stay aligned with the updated geometry. After an edit, the representation is directly reusable: the unchanged parts remain intact, and the modified regions are stored as local patches. This allows iterative editing with no global retraining.
The NEO-Dataset provides ground-truth RGB-D images and object poses for benchmarking. Metrics are computed separately for the “In” and “Out” masked regions, isolating the model’s performance on the most challenging parts of the edit: reconstructing surfaces that were previously unseen (under the removed object) and rendering the moved object at its new location. For the assembly experiment, only the initial scene and target motions are given; intermediate states are generated by NEO and compared to real captures.
Why This Matters for Robotics
Most robotic manipulation systems rely on repeated scanning to update their understanding of a changing workspace. NEO offers a path to a persistent, editable 3D model that evolves with each action, reducing the need for multiple re-scanning passes. This is especially valuable in assembly, pick-and-place, and bin-picking tasks where the scene changes rapidly.
By allowing language-prompted edits, NEO bridges the gap between human instruction and robot action. A worker can say “move the blue block to the left corner,” and the robot can visualize the result before executing. This capability is directly relevant to warehouse robots that must rearrange items, and to humanoid robots performing complex tabletop tasks. The fact that NEO supports sequential edits without retraining makes it practical for real-time operation, where computation budgets are tight.
Limitations and Open Questions
NEO assumes the initial NeRF is trained on a static scene with good coverage. If the initial scan misses large occluded areas, subsequent edits may introduce artifacts. The diffusion-based inpainting step, while effective, can sometimes produce minor inconsistencies under repeated edits, especially when the same region is modified several times. The method has only been validated on tabletop scenes with a few objects; scaling to cluttered industrial bins or full rooms remains untested. Additionally, language-based object selection works well for distinct objects but may struggle with ambiguous or partially occluded items.
Frequently Asked Questions
What makes NEO different from previous NeRF editing methods? NEO does not require retraining after each edit — it performs local updates that preserve the original structure, enabling dozens of sequential modifications from a single initial scan.
Can NEO handle object insertion, not just removal or movement? Yes. The same density-transfer and inpainting pipeline supports inserting objects from a library or copying objects within the scene.
What hardware is needed to run NEO? A single GPU (e.g., NVIDIA RTX 3090) is sufficient for editing a typical tabletop NeRF. The initial training takes about 20 minutes, and each edit takes under a minute.
Is the NEO-Dataset publicly available? Yes, the dataset includes 6 scenes with ground-truth before/after captures of object removal and manipulation, along with object poses from the YCB dataset.
Conclusion
NEO makes NeRF scene editing practical for real-world robotics by enabling unlimited sequential edits from a single initial model. It provides accurate predictions of manipulation outcomes and supports autonomous execution, closing the loop between perception and action. This work brings editable 3D scene representations one step closer to deployment in dynamic industrial environments.