Given a robot demonstration video, a language instruction, and gripper proprioceptive signal, SPARC produces object-centric spatial annotations for each interaction segment. Each annotation contains a subtask instruction, the manipulated object name, initial and target object locations, an object trajectory, gripper phase boundaries, and a scalar reliability score. SPARC proceeds in three stages: subtask decomposition, candidate proposal and tracking, and reliability scoring.
Method
SPARC operates in three stages. First, it decomposes the demonstration into subtasks by aligning the language instruction with gripper proprioceptive signals and video frames. Second, it proposes candidate object detections and tracks them across the relevant temporal segments. Third, it computes a reliability score for each candidate annotation using physically grounded motion cues.
Reliability Scoring with Motion Cues
The reliability score is computed by evaluating each candidate annotation against several interaction-aware criteria. These include the consistency between the object trajectory and gripper motion, the temporal alignment of grasp and release phases, and the spatial overlap between the object and gripper during manipulation. The score combines these cues into a single scalar value that indicates the trustworthiness of the annotation without requiring any human review.
Detector Confidence vs. Reliability Score
SPARC replaces the commonly used detector confidence with a physically grounded reliability score. Detector confidence alone is a poor indicator of annotation quality in manipulation tasks because it reflects only the certainty of the object detection, not whether the detected object is the one being manipulated. The reliability score captures task-relevant cues such as whether the object moves during the grasp phase and whether the gripper contacts it appropriately.
Robot-Overlap Filter and False Suppression
A hard filter that excludes candidates overlapping with the robot gripper can incorrectly suppress the correct annotation because the manipulated object often overlaps the gripper during contact. SPARC avoids this trade-off by replacing the hard gate with a graded quadratic penalty that down-weights overlapping candidates rather than excluding them. Combined with its other interaction-aware scoring components, this recovers falsely suppressed cases and raises accuracy to a significantly higher level.
IoU-Threshold Robustness
All methods lose a small amount of accuracy under tighter matching thresholds, but the loss is uniform across variants and the margin of SPARC over the strongest ablation is retained. This indicates that the improvements come from selecting the correct object rather than from boxes that only loosely overlap the ground truth, which would degrade quickly under stricter thresholds.
VQA Dataset Generation
To generate the VQA dataset, templated question-answer pairs are used across three task types. In object grounding, the VLM points to a specified object from either its name or the demonstration instruction. In vacant location pointing, the VLM points to the start or target location of the demonstration. In trajectory prediction, the VLM predicts the object trajectory for a given task instruction. The same templates and prompts are used for all variants, replacing only the underlying spatial annotations with either SPARC or detection-generated annotations.
For target point supervision, a point is sampled from the winner candidate mask. For vacant pointing, the extracted start or target location is used and the first or last observation frame in which the object is absent at that location is queried. Points are then sampled at the corresponding vacant location. This provides a proxy for pointing to a vacant location without requiring explicit location annotations. For trajectory tasks, five equally spaced points are subsampled from the object trajectory. All spatial coordinates are normalized to the range between 0 and 1.
Downstream Dataset Scaling Experiments
SPARC provides a stronger scaling signal than detector confidence on both the overall benchmark suite and the interaction-aware benchmark (IA-Bench). On the aggregate score, SPARC improves from 0.554 at 50K samples to 0.583 at 200K, then decreases to 0.563 at 500K and 0.557 at 838K. This suggests that scaling is beneficial until lower-ranked annotations enter a noisier data regime. However, on IA-Bench, SPARC continues to improve with scale, rising from 0.716 at 50K to 0.785 at 838K, close to the quality-filtered reference of 0.789. In contrast, detector confidence degrades as more samples are added in both settings, indicating that detector confidence increasingly admits noisy supervision while SPARC preserves useful supervision over a much larger annotation budget.
Per-Task Results
Reasoning vision-language-action (VLA) policies trained on annotations generated by SPARC consistently outperform baselines by a large margin on most tasks, highlighting the improved grounding and spatial localization capabilities resulting from higher quality spatial annotations.
Conclusion
SPARC is an automatic spatial annotation pipeline that scores candidates with physically grounded motion cues rather than detector confidence. The resulting reliability score makes the trade-off between quality and scale controllable without any human review. SPARC yields more accurate annotations than detection-based baselines and retains more usable data at a fixed precision target. Models trained on SPARC annotations surpass those trained on human-annotated data across object-grounding and motion-aware benchmarks, and reasoning policies trained on them are substantially more robust in cluttered real-world manipulation.
Limitations
SPARC is bounded by the off-the-shelf models it builds on: detector errors limit part-level annotations, tracker identity switches occur for visually similar objects, and 3D lifting struggles with transparent surfaces. It assumes a mostly static camera, so tracking degrades under strong motion, and relies on accurate phase intervals, letting upstream subtask-segmentation errors propagate downstream. SPARC is most reliable for tasks with distinct grasp phases.
Frequently Asked Questions
What makes SPARC's reliability score different from detector confidence? The reliability score uses physically grounded motion cues such as object motion during grasp phases and gripper-object contact, whereas detector confidence only reflects certainty about the object's presence in the image.
Does SPARC require human annotation or review? No, SPARC operates fully automatically. The reliability score enables controllable trade-offs between quality and scale without any human review.
What types of tasks is SPARC best suited for? SPARC is most reliable for tasks with distinct grasp phases where the manipulated object clearly separates from the gripper before and after contact.
How does SPARC handle objects that overlap with the robot gripper during contact? Instead of using a hard filter that excludes overlapping candidates, SPARC applies a graded quadratic penalty that down-weights overlapping candidates while still allowing correct annotations to be selected.