A new approach called GROW² enables robots to pick the right tool from a messy pile of everyday objects and figure out exactly where to grasp and apply it—all without prior training on specific tools. By combining vision-language understanding with 3D reconstruction, the system achieves tool-use success rates that significantly outperform current methods in both simulation and real-world tests.
What the Researchers Built
GROW² stands for “Grounding Which and Where for Robot Tool Use.” It tackles a problem that humans solve effortlessly but robots find notoriously hard: spotting an improvised tool in a cluttered scene and knowing exactly where to apply it. For example, if a knife is missing, a robot should be able to grab a plate and use its edge to cut a cake.
The system works in three stages. First, a vision-language model (VLM) scans the scene and picks the best candidate tool from the objects present. Second, the same VLM decomposes that tool into named parts (e.g., “edge,” “handle”) and selects the appropriate part for the task. Third, GROW² grounds that part name into precise 3D coordinates using a combination of segmentation and 3D reconstruction, producing a clear affordance region the robot can grasp and apply.
Crucially, GROW² does not assume the tool is already identified, nor does it rely on a fixed library of tools with predefined functions. It handles open-category objects—items the system has never seen before—by leveraging foundational models for semantic understanding and geometric reasoning.
Key Results
GROW² was evaluated on two standard affordance benchmarks and a new benchmark designed specifically for tool selection. On the AGD20K 2D affordance grounding dataset, GROW² achieved a mean intersection over union (mIoU) of 58.2%, outperforming the previous best method (AffordanceLLM at 53.1%). On the PIAD 3D point cloud benchmark, it reached a mean IoU of 68.7%, again beating state-of-the-art methods like GEAL (63.1%).
The researchers also created a new simulation benchmark with 5 task types, 85 unique object assets, and 100 scenes. GROW² succeeded in 72% of those trials, compared to 45% for the best baseline (MOKA). In real-world testing on a Franka arm with 50 trials, GROW² achieved a 64% success rate, while baselines ranged from 12% to 36%.
| Benchmark | Metric | GROW² | Best Baseline |
|---|---|---|---|
| AGD20K (2D affordance) | mIoU | 58.2% | 53.1% (AffordanceLLM) |
| PIAD (3D point cloud) | mIoU | 68.7% | 63.1% (GEAL) |
| Simulation (5 tasks, 85 objects) | Task success | 72% | 45% (MOKA) |
| Real-world (50 trials) | Task success | 64% | 36% (Best baseline) |
How It Works
GROW² is a modular pipeline that combines a large vision-language model (VLM), a 3D reconstruction tool (SAM3D), and a part-segmentation tool (SAM3). The process begins with a single RGB-D image of the cluttered scene. The VLM—in this case, a recent version of GPT—first enumerates all visible objects and selects the one that could best serve as a tool for the stated task (e.g., “cut the cake”). This step solves the which problem.
Next, the VLM decomposes the selected tool into its constituent parts and chooses the region appropriate for the action. For instance, a plate might be decomposed into “center,” “rim,” and “edge”; for cutting, the “edge” is selected. This part name is then used to prompt SAM3, a 2D segmentation model, to produce a pixel-level mask of that part across multiple rendered views of the object. The generated masks are back-projected into 3D using the object’s reconstructed mesh from SAM3D and registered point clouds.
To generate the final affordance map, GROW² projects the segmented part masks onto the 3D mesh, computing a dense score per vertex that indicates how “part-like” that vertex is. The region with the highest scores becomes the grasp or contact area for the robot. The target object (e.g., the cake) is processed similarly to identify where the tool should make contact.
The entire pipeline runs in about 16.6 seconds per scene on an NVIDIA H100 GPU, with parallelized reconstruction of tool and target objects to reduce latency.
Why This Matters for Robotics
The ability to improvise tools from everyday objects is a cornerstone of robust autonomous manipulation. In homes, warehouses, or disaster zones, robots rarely encounter perfectly arranged environments with dedicated task-specific tools. GROW² brings robots a step closer to human-like resourcefulness.
For warehouse operations, a robot could use a nearby box edge as a scraper instead of a specialized tool, reducing downtime. In home service robotics, a robot could cut food with a plate or hammer a nail with a heavy mug—adapting on the fly to whatever is available.
The modular design also means GROW² can be updated as better VLMs and segmentation models emerge, without retraining the entire system. This makes it a practical foundation for future commercial robots. Explore used cobots for sale that could be equipped with similar zero-shot tool-use capabilities, or browse warehouse robots that could benefit from this level of adaptability.
Limitations and Open Questions
GROW² is not yet fast enough for dynamic tasks: the 16.6-second pipeline latency makes reactive manipulation impossible. Failures are concentrated in the VLM stages—occasionally the model picks a wrong tool or part. Scaling to cluttered scenes with many objects also increases the chance of hallucination.
Efficiency is another concern. Running large VLMs and multiple foundation models is computationally expensive, which limits deployment on edge devices. The authors suggest distilling GROW² into a lightweight model as future work, but that would likely trade off some generalization ability.
Finally, the system relies on accurate 3D reconstruction; if the object mesh is incomplete or distorted, part grounding suffers. Handling transparent, shiny, or highly deformable objects remains an open challenge.
Frequently Asked Questions
What exactly does GROW² allow a robot to do? It enables a robot to look at a cluttered scene of unknown objects, pick the best item to use as a tool, and know precisely where to grasp and apply it—all without any prior training on those specific objects.
How is GROW² different from previous affordance learning methods? Earlier methods either assumed the tool was already known or relied on a fixed set of tools with predefined functions. GROW² chooses both the tool and the interaction region on the fly, handling objects it has never seen.
Does GROW² require a lot of annotated training data? No. It uses pre-trained vision-language models and segmentation models, so it requires zero additional annotations for the tool-selection task. The 2D and 3D affordance benchmarks were used only for evaluation, not training.
How fast is GROW² in practice? The current pipeline takes about 16.6 seconds per scene on an H100 GPU, which is too slow for real-time dynamic tasks. The researchers are exploring ways to make it faster.
Conclusion
GROW² takes a significant step toward practical, open-world tool use for robots by grounding both which object to use and where to interact. Its strong benchmark performance and real-world success suggest that combining general VLMs with 3D reconstruction is a promising path for adaptive manipulation.