Related Work
Autonomous Robot Data Generation
Data generators scale robot experience by retargeting object-relative demonstration segments across scene configurations and decomposing behavior into reusable skills connected by planned motions. These methods convert sparse demonstrations into diverse imitation data, yet task semantics remain bound to the originals, lacking an independent, grounded monitor to validate learned-policy rollouts.
Formulation Reliability
Rows C report the language interaction. Kuafu uses 110.5k tokens per task, compared with 234.4k for Eureka’s one-time reward search. VoxPoser instead uses 16.99 calls and 31.6k tokens per rollout.
At the reported mean rates, VoxPoser’s cumulative interaction exceeds Kuafu’s after just two executions, measured in calls, and four rollouts, measured in tokens. Thus, forming a persistent program converts language interaction from a recurring execution cost into a task-level cost.
Visual Learning and Zero-Shot Transfer
For physical deployment, each simulated setup reproduces the calibrated configuration of two or three RGB-D cameras corresponding to the workspace, with constrained randomization.
The dataset is doubled to 1,000 trajectories per task, and the resulting policies are deployed without real-world fine-tuning. DP3 achieves 36 successes in 106 trials across six tasks and three robot-gripper configurations without real-world fine-tuning. This yields a 34.72% task-macro success rate and nonzero success on every tested task.
The pooled success rate is 33.96%, with a 95% Wilson interval of 25.6–43.4%. The smallest task shows 2/6 successes, with an interval of 9.7–70.0%.
Inspection of failed trials identifies precise grasp acquisition as the main remaining challenge. T1 has the lowest success rate at 15%, and grasping is the most frequent observed failure mode on T7.
These results establish zero-shot sim-to-real feasibility using datasets produced under persistent SUN programs.
Conclusions
Scope and Limitations
SUN Programs rely on a registered scene interface with informative object frames and axes, as well as a finite, typed-operator library. Consequently, Kuafu cannot handle deformable objects or fluids without defining new operators.
The evaluation does not cover held-out instructions, operators, scene semantics, or task compositions, limiting strong claims of open-world generalization.
Methodologically, the stage monitor advances monotonically and cannot roll back after a physical regression. Additionally, the Q95 metric is model-predictive-control-relative, measuring efficiency against a specific planner rather than absolute trajectory quality.
Finally, validation across six tasks and 106 trials establishes zero-shot feasibility across the tested robot-gripper configurations but does not guarantee broad sim-to-real reliability in unstructured environments.
Frequently Asked Questions
What is the main purpose of persistent SUN programs? They convert language interaction from a recurring execution cost into a task-level cost while supporting learned-policy rollout validation.
How many tokens does Kuafu use per task? Kuafu uses 110.5k tokens per task.
How did DP3 perform during physical deployment? DP3 achieved 36 successes in 106 trials across six tasks and three robot-gripper configurations without real-world fine-tuning.
What is the main remaining challenge identified in failed trials? Precise grasp acquisition is the main remaining challenge, with grasping the most frequent observed failure mode on T7.