We study Guided Data Collection (GDC): rather than collecting data indiscriminately and filtering it afterward, GDC methods actively reason about what data to collect, where to collect it, and which interactions to prioritize to maximize downstream deployment performance. We focus on the practical setting where direct access to the deployment environment is unavailable, but data collection remains possible in related auxiliary environments.
Task-Parameter-Space Exploration Through Commonsense-Grounded Proposals
Given target-domain context—video demonstrations, language descriptions, and available training assets such as environments and objects—the vision-language model (VLM) proposes feasible task parameters specifying the collection environment, scene configuration, and manipulation task. The VLM does not determine how much data should be collected from each task; instead, it identifies promising regions of the open-ended task space, while attribution-based evaluation later determines their utility.
Experiments
Training Details
We train a language-conditioned Octo policy on the collected datasets, initializing from the pretrained checkpoint released by the authors. Following prior work, we co-train the policy with the target-task demonstrations using a 50/50 sampling mixture.
Qualitative Analysis: What Data Does GUILD Collect?
The final data composition for each real-world target task collected by GUILD in the Open-World setting shows that, for each target, the auxiliary tasks are unique to that setting and ordered from highest to lowest collected data. Each bar denotes the number of trajectories collected for an auxiliary task, while its color indicates the outer-loop iteration in which that task was introduced. Representative high-allocation auxiliary tasks are also shown for each target.
GUILD prioritizes auxiliary tasks that preserve relevant semantic and behavioral structure with the target task, such as similar objects, receptacles, or manipulation patterns.
Conclusion and Limitations
This work was conducted in the Robot Interactive Intelligence (RobIn) Lab at The University of Texas at Austin and supported in part by the National Science Foundation (CAREER Award FAIN 2543745), the NVIDIA Academic Grant Program and TACC resources, and Amazon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation or the other sponsors.
Human Experiment Details
To evaluate human-guided data collection, we surveyed participants with varying levels of robotics experience. The least experienced participant had approximately one year of robotics experience, while the most experienced participants had more than five years of research experience in robot learning and manipulation. To construct human-proposed datasets, we provided participants with the same information available to the automated methods, including the target-task context, descriptions of the available training environments, and the total data collection budget. Participants were instructed to design a data collection strategy that would maximize performance on the target task while respecting the specified budget constraints. Importantly, participants did not have access to attribution scores, baseline results, or any additional feedback beyond the initial information provided to the automated methods.
For each target task, participants proposed a set of tasks within the same parameter space used by GUILD. For example, in the Real Open-World setting, each proposal specified a collection environment, object configuration, task description, and the amount of collection budget allocated to that task. The proposed datasets were then collected using the same data-collection process employed for the automated methods: scripted policies in simulation and human teleoperation in the real world. Finally, all human-guided datasets were evaluated using the identical policy training and evaluation protocol described for each setting. This ensures that differences in performance arise solely from the data collection strategy rather than the learning or evaluation procedure.
Impact of the VLM Used
Throughout our experiments, we use GPT-5.2 for task proposal and feedback-driven support expansion. We study the sensitivity of GUILD to the choice of VLM by replacing GPT-5.2 with Gemini-2.5-Flash in our LIBERO experiments, while keeping all other hyperparameters fixed. The results are reported as GUILD (Gemini).
Stage-Wise Results for Real-World Tasks
We measure stage-wise success rates for the real-world evaluations. The stages for each task are listed below:
- Pink-cup-sink: Pick up the pink cup → place it in the sink.
- Towel-drawer-close: Pick up the towel → place it in the open drawer → close the drawer.
- Bowl-pour: Grasp the black bowl → pour its contents into the drawer.
- Lemon-drawer-close: Pick up the lemon → place it in the open drawer → close the drawer.
Frequently Asked Questions
What problem does GUILD address? GUILD studies guided data collection, in which a method reasons about what data to collect and where in order to improve downstream deployment performance.
What information does the VLM use to propose tasks? It uses target-domain context, including video demonstrations, language descriptions, and available training assets such as environments and objects.
Does the VLM decide how much data to collect for each task? No. It proposes promising regions of the task space; attribution-based evaluation later determines their utility.
How were human-guided datasets evaluated? They used the same policy training and evaluation protocol as the automated-method datasets, so performance differences reflect data collection strategy.