Overview of RAPID
Program Representation
RAPID introduces an object-centric relational program representation that enables strategies constructed by the agent to generalize across different scenes. RAPID represents each primitive as a geometry- and physics-aware local trajectory-optimization program that achieves an object-level effect.
A strategy then composes these primitives through relational constraints that specify the desired effect of each primitive.
Strategy Deployment
At deployment, given an RGB-D observation of a novel scene and a language instruction, RAPID reconstructs the scene in simulation, instantiates the strategy in the reconstructed scene, and executes the resulting trajectory on the robot.
Lightweight semantic binding assigns scene objects to the semantic roles in the strategy, while the strategy and primitives themselves remain unchanged.
Object-Centric Relational Programs
RAPID defines an object-centric relational program representation consisting of primitives, their composition into task-level strategies, and strategy instantiation in novel scenes.
Strategy Instantiation in Novel Scenes
Throughout this process, the strategy program and the referenced primitives remain frozen.
The strategy program remains fixed while its primitives are adapted to the new scene.
Generalization therefore arises from the object-centric relational program representation, together with the adaptation of trajectory optimization to the geometry and physical properties of the new scene.
Reconstructing Interactive Environments
Feasible Scene Variants
To construct variants of the demonstrated scene, RAPID uses a task-agnostic variation sampler to randomize object appearances, sizes, poses, and physical parameters such as mass, friction, and elasticity.
However, the randomly sampled scenes might not be feasible for a certain primitive or strategy. Therefore, RAPID asks the coding agent to write a filter for each primitive or strategy that accepts only variants preserving the declared roles and relations, task meaning, and physical feasibility.
Real-World Experiments for Nonprehensile Manipulation
Results
With the object-centric relational program representation and compliant operational-space controller, contact-rich nonprehensile behaviors optimized in simulation can be transferred directly to the real robot and executed effectively.
In contrast, CaP-Agent0 struggles to generate reliable trajectories using procedural programs.
Moreover, leveraging the vision-language model’s prior knowledge to estimate physical parameters such as mass, friction, and restitution, together with SAM 3D’s strong 3D reconstruction capability, helps keep the real-to-sim and sim-to-real gaps manageable.
Conclusion
RAPID is a framework that closes the agentic coding loop for robotic manipulation from a single visual human demonstration and a language description.
By automatically deriving task specifications, manipulation primitives, and interactive verification environments, RAPID enables a coding agent to construct, execute, and iteratively refine reusable manipulation strategies.
Its object-centric relational program representation represents manipulation primitives as trajectory-optimization programs and composes them with relational constraints, enabling generalization to novel scenes.
Experiments on eight contact-rich nonprehensile manipulation tasks demonstrate strong generalization across object and scene variations, while evaluations on LIBERO-Pro and a real robotic system support RAPID’s broader applicability and practical deployment.
In future work, RAPID will incorporate other tools for learning primitives, such as reinforcement learning for manipulation with dexterous hands. Further, RAPID will automatically benefit from continued progress in coding agents, robot foundation models, and simulation engines, including those for deformable objects, and scale up.
Frequently Asked Questions
What is RAPID? RAPID is a framework for robotic manipulation that uses a coding agent to construct, execute, and refine reusable strategies from a visual human demonstration and a language description.
How does RAPID generalize manipulation strategies to new scenes? It uses object-centric relational programs, lightweight semantic binding, and trajectory optimization adapted to the new scene’s geometry and physical properties.
What physical parameters can RAPID estimate? RAPID can estimate parameters such as mass, friction, and restitution using prior knowledge from a vision-language model.
How many manipulation tasks were evaluated? The experiments evaluated eight contact-rich nonprehensile manipulation tasks, along with tests on LIBERO-Pro and a real robotic system.