Related Work
Structured Visual Prompting
A complementary line reformats the scene into 2D inputs that multimodal large language models already parse well. Set-of-Mark prompting overlays visual anchors on images; GPT4Scene pairs video frames with a global bird’s-eye-view image; and Struct2D pairs a marked bird’s-eye view with object metadata, in both zero-shot and fine-tuned forms.
These schemes apply one fixed rendering to every question, whereas GraFT’s bird’s-eye-view module renders on demand. Each view is conditioned on the question and shows only the referenced objects.
Tool-Augmented Spatial Reasoning
A further line grants the model access to external tools. SpatialScore’s SpatialAgent, TIGeR, and SpaceTools let a multimodal large language model invoke estimators for depth, pose, or bounding boxes, typically rerunning perception for each question.
GraFT runs perception only once, during graph construction. Its reasoning-time tools then perform deterministic geometry over the graph, amortizing perception across all questions.
Symbolic Module: Geometric Tools
Each object’s box encodes its position, size, and orientation in three-dimensional space to the precision of the upstream perception. Any quantity that is a function of this geometry then follows deterministically from the boxes, so its accuracy is inherited from the perception rather than re-estimated by the backbone.
Inferring such quantities from an image would only add error on top. GraFT therefore computes them directly from the 3D scene graph through symbolic geometric tools that read straight from the boxes:
- A count tool lists each object class with its instance count.
- A size tool returns an object’s width, depth, and height as the extents of its box.
- A distance tool measures the surface-to-surface distance between the boxes of two object classes.
- A proximity tool ranks several candidate classes by their closest distance to an anchor object, resolving which is nearest or farthest.
- A room tool returns the room’s width, depth, height, and floor area.
Each tool runs deterministically over the boxes, so no part of the numerical computation passes through the backbone.
Resolving a geometric task takes exactly one tool call. GraFT never chains tools or feeds one tool’s output into the next, keeping the symbolic module a single deterministic step.
GraFT Versus Prior Work
In the last experiment, GraFT is tested on whether it can match models trained for spatial reasoning. It is compared head-to-head with Struct2D on the subsets it reports and on the complete VSI-Bench under realistic perception.
The referenced tables are not included in the provided paper text.
Conclusion
GraFT is a training-free framework that gives a frozen multimodal large language model three-dimensional spatial reasoning by operating over a single 3D scene graph.
From this 3D scene graph, GraFT derives three complementary capabilities:
- Symbolic geometric tools for metric estimation.
- A selectively rendered bird’s-eye view for allocentric layout.
- Geometry-ranked egocentric frames for visual attributes.
Each capability supplies the backbone with the evidence its task demands. Per-module studies under both ground-truth and realistic perception settings validate each capability on its target tasks.
GraFT lifts three open-source backbones by 37% to 65% on the VSI-Bench average, reaching up to 51.4 and surpassing several models fine-tuned for spatial reasoning. Even a 3B backbone overtakes the strongest 72B baseline.
These results show that a frozen multimodal large language model paired with a compact 3D scene graph is already a strong spatial reasoner, without spatial supervision or a backbone-specific encoder.
Its main limitation is that accuracy is bounded by scene-graph quality. Since this compact graph is easy to maintain and shared by every capability, new modules over it can extend GraFT to further spatial tasks.
3D Scene Graph Construction
Under ground-truth perception, the graph is built from the dataset annotations of ARKitScenes, ScanNet, and ScanNet++.
Under realistic perception, GraFT follows the Struct2D setup. The scene is reconstructed offline, objects are detected with UniDet3D and Mask3D, and the results are converted into the same graph structure as the ground-truth 3D scene graph. This graph serves as the interface for the reasoning modules.
Per-frame camera poses and intrinsics are kept with the graph for the egocentric module.
Frequently Asked Questions
What is GraFT? GraFT is a training-free framework that enables a frozen multimodal large language model to perform three-dimensional spatial reasoning over a single 3D scene graph.
What capabilities does GraFT derive from its scene graph? It provides symbolic geometric tools, selectively rendered bird’s-eye views, and geometry-ranked egocentric frames.
How does GraFT handle geometric tasks? It computes quantities deterministically from object boxes using exactly one symbolic tool call, without chaining tools.
What is GraFT’s main limitation? Its accuracy is bounded by the quality of the constructed scene graph.