A Gaussian-native pipeline preserves unedited captured appearance while turning selected objects into independently movable assets. Coupled Scene Construction connects candidate alignment and selection to source-Gaussian removal and background completion. Evaluation on real scenes spans construction availability, geometric and visual fidelity, physical validity, and manipulation performance.
Introduction
The contributions are:
- A Gaussian-native pipeline that preserves unedited captured appearance while turning selected objects into independently movable assets.
- Coupled Scene Construction, in which shared object evidence connects candidate alignment and selection to source-Gaussian removal and background completion.
- A real-scene evaluation spanning construction availability, geometric and visual fidelity, physical validity, and manipulation performance.
Scene Observation
From the captured Gaussians and calibrated RGB and pose inputs, Scene Observation supplies object masks and the observed surface. Frozen SAM3 produces image masks, which are lifted onto the scene surface using first-hit ray intersections. These regions are associated across views by voxel overlap to obtain object instances with surface samples, masks, and spatial bounds.
Coupled Scene Construction
The two branches share object identity but complete different missing content: object shape and the exposed background. A registered asset also supplies shape support for source-Gaussian removal.
The two directions penalize unsupported candidate surfaces and unexplained observations. Clipping at [a distance threshold that limits outliers] limits outliers, as do penalties on unobserved regions. This measures observed agreement, not hidden-shape correctness.
Verify, Select, and Retry
Candidates are ranked using construction checks for registration, scale, observation support, collision validity, and isolated settling, followed by residual and stability criteria. If initial candidates fail, a bounded retry tests alternative source-up directions. It changes the alignment hypothesis without regenerating or repairing shape.
The default constructor keeps the best numerically valid candidate and records unresolved checks. A strict variant rejects candidates that still fail, separating the effects of registration retry from rejection.
Interactive Environment
Registered assets and background Gaussians form the interactive scene.
Collision meshes determine contacts, the simulator computes body poses, and Gaussians provide the corresponding visual appearance. Where supported, CoACD converts completed meshes into convex collision components. Physical parameters come from priors unless separately measured.
This applies the body’s relative motion while preserving the initial body-to-asset offset, without requiring the visual origin to coincide with the center of mass. Shared motion does not guarantee identical visual and collision surfaces.
For scale [the asset scale factor], rotation [the asset rotation transform], and translation [the asset translation vector], Gaussian means and covariances transform accordingly.
Scale is applied once at construction, after which body motion is rigid. Cameras, bodies, and renderings use the same simulator state. View-dependent appearance requires consistent viewing directions.
Here, simulation-ready means loadable rigid objects with state-linked appearance, not a guarantee of physical accuracy or task utility.
Experimental Setting
For [the number of planned trials] planned trials, [the number of executions] executions, and [the number of successes] successes, construction rejections contribute no successes to fully accounted studies, while missing evaluations remain unmeasured.
Policy intervals use paired hierarchical resampling over layouts, configurations, and resets, rather than treating resets as independent reconstructions.
Qualitative Results
Coupled Object and Background Construction
The conversion proceeds from a captured Gaussian scene through source removal and background inpainting to a registered asset with physical parameters. The sequence illustrates that the inserted asset corresponds to the source appearance being removed, while completion supplies the background exposed by that removal. The parameter panel reports assigned physical properties.
Appearance Harmonization
Rendered observations are compared with and without the optional Harmonizer. Unlike background completion, which constructs scene Gaussians, this stage modifies rendered images without changing geometry or simulator state. It addresses the appearance mismatch discussed in the paper.
Discussions
Component Evidence
Experiments test selection, registration retry, and backend choices, while the input study measures construction quality, not discovery precision or recall. Source-Gaussian removal, local 3D fill, and simulator-linked appearance have qualitative evidence, but no isolated quantitative tests.
Interaction Scope
In a separate 48-configuration study with shared repair actions, adaptive and fixed-order contextual repair both yield 65/480 successes versus 121/480 without repair. Conditional success rises from 37.8% to 54.2%, but executions fall from 320 to 120.
Separately, destination reconstruction reduces successes from 45/240 to 20/240 across 24 configurations and four layouts, retaining native articulation and goals. These results expose coverage and asset-compatibility limitations.
Contextual repair differs from alignment-only retry, and policy success does not establish real-world dynamic fidelity.
Capture and Physical Scope
The exploratory TRELLIS-only RGB-video study yields seven loadable targets from 24 instances and 21/70 successes versus 53/70 for the reference, with 170 method trials unmeasured. Association, calibration, and appearance issues, plus a post-failure marker-estimator revision, limit interpretation.
The pipeline assumes rigid objects, prior physical parameters, and locally planar fill. Articulation, deformables, identified dynamics, arbitrary hidden geometry, and physical-robot transfer remain outside the demonstrated scope.
Conclusion
We presented φ-RIE, a Gaussian-native pipeline that converts selected objects into movable simulator assets while preserving unedited scene content. Shared object evidence couples asset construction, source removal, and background completion, while simulator poses drive Gaussian appearance.
Experiments show improved geometry at fixed retention and manipulation gains over a single-generator baseline in controlled asset-replacement tests, alongside a visual-fidelity trade-off. This coupling extends captured Gaussian scenes beyond static rendering to object-level interaction.
Generative AI Use Disclosure
OpenAI Codex was used to assist with drafting and debugging portions of the experimental code and with language editing throughout the manuscript. The authors reviewed and validated the resulting code and text.
All reported results were obtained from actual experiment runs and verified by the authors; no empirical result values were invented, altered, or synthesized by AI.
Frequently Asked Questions
What is φ-RIE designed to do? It converts selected objects in captured Gaussian scenes into movable simulator assets while preserving unedited scene content.
How does Coupled Scene Construction use shared object evidence? It connects candidate alignment and selection with source-Gaussian removal and background completion.
What does simulation-ready mean in this work? It means the scene contains loadable rigid objects with state-linked appearance, not that physical accuracy or task utility is guaranteed.
What limitations remain outside the demonstrated scope? Articulation, deformable objects, identified dynamics, arbitrary hidden geometry, and physical-robot transfer remain outside the demonstrated scope.