The vision of generalist robotics is to develop robots that perform a broad range of tasks reliably in complex real-world environments, where objects, layouts, and instructions keep changing, while improving through experience. This shifts the goal of robot agents from episode-level task completion toward sustained capability accumulation: failed grasps, stale object locations, and recovery situations are valuable only if the robot can identify the capability involved, refine it using execution evidence, and make the resulting behavior available to later tasks.
Execution experience, however, becomes a reusable capability only when it is organized around the task structure that gives it meaning. A repair must remain connected to the objective that motivated it, belong to the capability responsible for the observed behavior, and be validated before it is reused. Without such structure, repeated attempts fill the context with complete trajectories, local fixes drift from the overall objective, and people are drawn back into resetting environments, reading logs, and deciding where each fix belongs.
Our observation is that this structure must serve both the people directing the robot and the agents implementing its behavior, which calls for two complementary abstractions. Human-friendly steering lets people express objectives, domain knowledge, constraints, and safety judgments at the level of the service they need. Agent-friendly structure gives agents task-relevant context, explicit interfaces, and bounded responsibilities for execution and revision.
Based on this observation, we introduce RoboRSI, a multi-agent system for robot self-improvement organized by Top-Down Skill Refinement (TSR). TSR decomposes each task into compound, atomic, and base skills with scoped responsibilities and explicit input–output contracts, attributes each execution outcome to the responsible branch, and confines the revision to that branch: an inadequate decomposition revises the task structure, an unreliable composition the corresponding compound skill, and a failed operation the responsible atomic or base skill. A Manager, Planner, Engineer, and Reviewer carry out planning, execution, diagnosis, and validated release, and stable skill sequences are consolidated into compound skills that later executions select and monitor instead of reconstructing every tool decision.
We evaluate RoboRSI on a mobile manipulator that performs multi-object household cleanup across a scene change while its skills are developed over 104 rounds, and in simulation against robot-agent baselines on LIBERO, LIBERO-PRO, and RoboTwin, with frozen capability transfer to LIBERO-Plus. RoboRSI achieves the highest success rate on all four benchmarks, exceeding the strongest baseline by 5.3 points on LIBERO, 11.0 on LIBERO-PRO, 5.7 on LIBERO-Plus, and 2.7 on RoboTwin.
Our contributions are threefold:
- Complementary abstractions for robot self-improvement: Execution experience becomes reusable only when tied to the task structure that gives it meaning. This structure must support both human-friendly steering and agent-friendly organization.
- RoboRSI framework: The system realizes these abstractions through Top-Down Skill Refinement, which verifies each atomic task’s postcondition before completion and revises each failure at its earliest failing node, and a multi-agent execution–refinement loop that validates and releases skills across tasks.
- Real-world and simulated evaluation: Skill development on a mobile manipulator over 104 rounds, the highest success on four simulation benchmarks, and ablations of role separation, skill refinement, and compound skills.
Method
Consider a coding agent connected directly to robot tools and asked to improve a household cleanup task in a complex real-world home over many runs. The context fills with records of earlier runs; a failure observed late in a sequence is patched where it appears, for example by adding retries to the release step, although it originated in perception or navigation; and because the objective is implicit in the latest prompt, successive edits optimize for the latest scene. People are then drawn back into reading logs and deciding where each fix belongs. RoboRSI removes this burden by making both the intent of the people and the work of the agents explicit.
Overall Framework
RoboRSI turns repeated robot execution into a growing library of reusable skills, organized around the two abstractions described above. Human-friendly steering is the channel through which people enter objectives, constraints, domain knowledge, and corrections at the level of the task; each correction is kept as maintained guidance on the responsible skill. Agent-friendly structure is the task–skill hierarchy in which the agents work: every task and skill has an interface, a declared responsibility, and a record of recent executions, so each agent receives only the part of the hierarchy that its current step concerns.
The framework’s development round proceeds as follows: agents decompose a stated goal and plan over released skills, execute them down to base skills, locate the earliest failing node from visible evidence and the tool trace, and validate and release the resulting updates for the next round. People steer at the first stage and whenever a correction is needed.
Top-Down Skill Refinement
Task–skill structures during physical task development
Two TSR revisions on the physical robot
Experiment
We report physical deployment and simulation results, then compare individual episodes and summarize the ablations.
Physical Robot Case Study
We study household floor cleanup with a WheelSingleArm M1 mobile manipulator. The robot searches for objects, approaches and grasps them, and transports them to designated receptacles. The case study covers 104 physical development runs with 24 hours of cumulative operation. Before each run, the scene is restored to a common base layout, while the objects and receptacles are partially randomized; the navigation map remains fixed. Platform details and the reset procedure are given in the appendix.
Case Study
We examine three episodes in which RoboRSI succeeds while the baselines fail from the same initial state, and then follow a revised skill as it is reused on tasks it was not developed on.
On LIBERO-Plus, the task is to place a cream-cheese package in a basket. OpenETA’s combined pick-and-place tool returned that the object had been released at the destination, and the agent declared success at once. CaP-X’s placement also reported a release while its containment check remained unverified. Both episodes failed. RoboRSI, with its frozen source library, received a placement result stating that the gripper had not opened. Rather than ending, it checked whether it still held the package, re-observed the basket from the head and top views, placed the package again, and succeeded.
Ablation Study
We study role separation, hierarchical refinement, code consolidation, and backbone choice separately, and additionally examine policy learning from execution data. The backbone study uses LIBERO-PRO and RoboTwin task panels with three evaluation episodes per task, 360 and 150 episodes per backbone.
We compare development on the 50 RoboTwin tasks with a single agent and with separate roles. In the single-agent condition, one Engineer plans, executes, and judges its own attempts. In the multi-agent condition, a Planner composes the plan, the Engineer executes it, and a Reviewer diagnoses each outcome from observations and tool traces. The single agent solves 9 of the 50 tasks, whereas the multi-agent system solves 36. Separating diagnosis from execution gives every failure an independent review before the next attempt, instead of leaving the executing agent to judge its own result.
Conclusions and Future Work
RoboRSI combines Top-Down Skill Refinement and multi-agent coordination to organize execution, diagnosis, local revision, and reusable capability, toward stable, efficient, and reusable robot self-evolution in complex real-world environments. Future work will transfer simulation-acquired experience to physical robots and share skills across robots and platforms.
Exploratory Development
Early LIBERO-Plus runs also allowed capability updates on the target benchmark. Their configurations allowed repeated attempts and target-side updates, whereas the main LIBERO-Plus transfer panel keeps source capabilities fixed.
Physical Skill Revisions in Rounds 25 and 33
Both records come from the household cleanup task on the mobile manipulator. Each revision was admitted with its trigger run, the four most recent eligible runs as a replay buffer, and the changed files; live iteration was paused until validation finished.
Frequently Asked Questions
What is RoboRSI? RoboRSI is a multi-agent system for robot self-improvement that organizes execution, diagnosis, skill revision, and validated release.
What does Top-Down Skill Refinement do? TSR attributes execution outcomes to the responsible task or skill and confines revisions to the earliest failing part of the hierarchy.
How was RoboRSI evaluated on a physical robot? A WheelSingleArm M1 mobile manipulator performed household floor cleanup across 104 development runs, with 24 hours of cumulative operation.
How did the multi-agent setup compare with a single agent on RoboTwin? The single agent solved 9 of 50 tasks, while the multi-agent system solved 36.