LLM Agents for Robot Manipulation
The knowledge these systems carry across episodes is textual or programmatic. RV-ICL keeps the harness unchanged and adds a visual record of how the task is performed, together with a protocol for consulting it.
Learning from Demonstrations in Context
In language models, a few demonstrations in the prompt suffice to specify a task, although models draw mostly on their format and label space rather than the exact input–label mapping. Flamingo extends in-context learning to interleaved images.
All of these present the demonstration once, as a prefix for the next actions. Our agent makes tens to hundreds of decisions per episode, so the question is not only whether to show a demonstration but which part, and when.
Recursion in Language-Model Reasoning and Retrieval
Recursion lets a bounded model handle inputs and problems that exceed its context. Recursive summarisation condenses a book into a tree of summaries read top-down, and RAPTOR retrieves from such a tree at the level of abstraction a question requires. For long videos, VideoAgent and VideoTree likewise inspect only the frames or branches a question needs.
Recursive decomposition splits a problem as needed. ADaPT decomposes a sub-task further only when the executor fails on it, THREAD spawns child threads that return only what the parent needs, and Recursive Language Models call themselves over slices of a context too long to read at once. Tree of Thoughts searches over such trees of intermediate steps.
A related family decides when to fetch information. FLARE retrieves when upcoming tokens are uncertain, and Self-RAG learns to request retrieval on demand. Reflexion and Self-Refine feed the outcome of an attempt into the next one, and MemGPT pages information between a bounded context and external storage.
RV-ICL brings recursion to an embodied agent. The demonstration is a tree read at the level each decision requires, and the agent descends whenever its progress in the physical task calls for more detail: before a delicate contact, at a new sub-goal, or after a step goes wrong.
The Demonstration Hierarchy
A fixed number of frames span the whole task. They are placed around every grasp and release, and uniform samples fill the rest. Each keyframe shows the third-person and wrist views and is captioned with its time fraction, gripper state, end-effector position, and event label.
Recursive Access
Before its first action, the agent reads the available hierarchy levels. It writes its sub-goals from the phase map, with the keyframes as the picture of each stage. It loads no clip yet, because the prompt asks for clips at the point of action rather than up front.
Main Results
While keeping the other five cells at or near the ceiling, RV-ICL brings clear gains on the three cells where HarnessVLA is weakest. Goal-T rises from 88.0 to 100.0, L10-T from 85.0 to 90.0, and L10-S from 72.0 to 84.0.
LIBERO-10 is the long-horizon suite of the benchmark, and each of its tasks chains several grasps and releases. The gains there suggest that recursive access is especially effective on such tasks.
The small drop on Goal-S (98.0 vs. 99.0) comes from a single failed episode. With five seeds per task, one episode moves a cell by two points. So the drop does not indicate that RV-ICL is worse than HarnessVLA on Goal-S.
How Agents Recurse
After the failed grasp, the agent asks for the grasp moment alone and reads from the clip how the demonstrator holds the pot: by the handle, with the fingers farther out than its own. Where that handle lies in its own scene, it determines from its own cameras, since the demonstration was recorded in another layout. The clip thus supplies what text memory cannot record, and nothing else from the video enters the context.
Across all 400 LIBERO-PRO episodes, the agent opens a few clips per episode, and almost every episode contains a re-entry.
In every episode the agent reads the keyframes and the phase map before its first robot action.
Limitations
RV-ICL assumes one demonstration per base task; when a benchmark rewrites the goal of a task, parts of the demonstration become invalid and must be flagged to the agent. On LIBERO, we detect sub-events from the gripper and from fixture joints. Tasks whose sub-tasks leave no such trace in the recorded state, such as pushing, wiping, or pouring, need another detector, for example one that reads the end-effector trajectory or asks a vision-language model. When contact is executed by a learned skill whose motion differs from the demonstrator’s, the finer levels carry little that the agent can act on.
Each demonstration view stays in context for the rest of the episode, so cost grows with the number of re-entries. All our experiments use LIBERO tasks with a single robot and a single VLA, so other embodiments and demonstration sources remain untested. Our agents are proprietary models accessed through third-party endpoints, whose behaviour may change over time.
Our LIBERO-PRO cells hold 50 episodes and each LIBERO-Plus dimension 17 or 18 variants on one seed, so a single episode moves a cell by two to six points. The baselines in the results table come from separate runs, so they cannot be compared with ours episode by episode.
Ethics Statement
All experiments are conducted in simulation. The LLM agents run shell commands and could read files outside their task; we isolated conditions by restricting directory permissions and audited every shell command afterwards. Deploying such agents on physical robots requires stronger sandboxing and human oversight. Demonstrations are taken from public benchmark datasets.
Frequently Asked Questions
What does RV-ICL add to an agent? It adds a visual record of task demonstrations and a protocol for consulting parts of that record as needed.
When does the agent retrieve video clips? It requests clips at the point of action rather than loading them all before acting.
What information does a retrieved clip provide after a failed grasp? It can show how the demonstrator performed the grasp, while the agent uses its own cameras to locate the object in its scene.
What are key limitations of RV-ICL? It assumes one demonstration per base task, and its experiments cover only LIBERO tasks with a single robot and a single VLA.