We presented Recurrent Generative Replay (ReGen), the first continual learning framework to leverage the generative capabilities of World Action Models (WAMs) as a native replay mechanism. By synthesizing pseudo-demonstrations conditioned on prior task instructions, current visual observations, and its own generated observations, ReGen mitigates catastrophic forgetting without requiring access to stored demonstrations from previous tasks. Experiments in both simulation and real-world manipulation demonstrate its effectiveness for continual robot learning. Finally, our analysis identifies long-horizon degradation in generated observations as the primary bottleneck, highlighting an important direction for future advances in WAMs and generative replay.
Method
We first present the continual learning problem formulation and then introduce our proposed framework for continual adaptation of WAMs, Recurrent Generative Replay (ReGen).
Problem Formulation
The continual learning setting involves training a robot policy sequentially across multiple tasks. At each stage, the agent observes a new task and must learn to perform it while retaining knowledge of previous tasks without storing their original demonstration data.
Recurrent Generative Replay (ReGen)
ReGen operates by having the current policy generate pseudo-trajectories for previous tasks using the WAM's generative capabilities. The policy conditions on prior task instructions, the current visual observation, and its own generated future observations to create synthetic demonstrations that serve as replay data. This approach eliminates the need for storing any prior-task demonstrations while enabling the policy to practice previously learned behaviors during new task training.
Experimental Results
We conduct experiments in both simulation and real-world environments to evaluate the effectiveness of the pseudo-trajectories generated by ReGen. We further perform representation analyses to characterize the quality of the generated trajectories and assess their deviation from perfect demonstrations.
Simulation Benchmarks
Experiments were conducted on the LIBERO benchmark suite, which provides a diverse set of manipulation tasks with varying levels of complexity. After each continual learning stage, we evaluate the policy on all tasks observed up to that point. For each task, we run a number of trials with randomized initial states and report the average success rate. From these per-task success rates, we compute the three continual learning metrics: Forward Transfer (FWT), Negative Backward Transfer (NBT), and Area Under the Curve (AUC).
Real-World Robot Evaluation
We evaluate on three real-world manipulation tasks introduced sequentially, with a number of trials per task from randomized object placements and initial gripper configurations. Rollouts are scored using a partial-scoring rubric: points for touching the target object and additional points for reaching the goal.
ReGen Analyses
We further analyze how ReGen preserves the policy's internal representations and behaviors throughout continual learning and study two key design choices in pseudo-trajectory generation.
Representation Preservation
Analysis of the learned feature spaces shows that ReGen maintains more consistent internal representations for previous tasks compared to standard sequential fine-tuning. The generated pseudo-trajectories help anchor the policy's understanding of earlier skills while accommodating new task knowledge.
Design Choices in Pseudo-Trajectory Generation
We investigate two critical design choices: the frequency of replay generation and the observation horizon used for conditioning. Results indicate that more frequent replay cycles improve retention but increase computational overhead, while longer observation horizons improve generation quality up to a point before degradation occurs.
Limitations of ReGen and Future Direction
Although ReGen substantially mitigates catastrophic forgetting without access to previous-task data, a performance gap remains relative to the privileged experience replay baseline. We identify the primary bottleneck as the limited generative fidelity of current WAMs.
Long-Horizon Degradation
Our analysis identifies long-horizon degradation in generated observations as the primary bottleneck. As the number of consecutive timesteps in generated trajectories increases, the visual quality and task-relevant content of the synthetic observations deteriorates, limiting the effectiveness of replay for longer-horizon tasks.
Future Directions
This finding highlights an important direction for future advances in WAMs and generative replay. Improvements in generative model architectures that maintain fidelity over longer horizons, or adaptive replay strategies that focus generation on critical timesteps, could further close the gap with oracle-based replay methods.
Conclusion
We presented Recurrent Generative Replay (ReGen), the first continual learning framework to leverage the generative capabilities of World Action Models as a native replay mechanism. By synthesizing pseudo-demonstrations conditioned on prior task instructions, current visual observations, and its own generated observations, ReGen mitigates catastrophic forgetting without requiring access to stored demonstrations from previous tasks. Experiments in both simulation and real-world manipulation demonstrate its effectiveness for continual robot learning.
Frequently Asked Questions
What problem does ReGen solve in robot learning? ReGen addresses catastrophic forgetting in continual robot learning, where a policy trained sequentially on multiple tasks loses performance on earlier tasks.
How does ReGen generate pseudo-trajectories without prior-task data? It uses a World Action Model conditioned on task instructions, current observations, and its own generated future observations to synthesize replay data for previous tasks.
What is the main limitation identified in the paper? Long-horizon degradation in generated observations is the primary bottleneck, where visual quality and task relevance decrease as trajectory length increases.
How is ReGen evaluated in real-world settings? The framework is tested on three sequential real-world manipulation tasks with randomized object placements, using a partial-scoring rubric for success evaluation.