Dense progress rewards and rich language feedback produced by D&A can be leveraged across downstream policy-improvement paradigms. This section outlines how D&A accelerates online and offline reinforcement learning (RL) and Code-as-Policies (CaP), the paradigms used in the experiments. D&A is a general-purpose reward and rich-feedback framework that can also be paired with other applications.
Using D&A for Downstream Policy Improvement
Code-as-Policies
Recent observations in robot failure analysis and feedback-driven optimization suggest that rich language feedback can improve policy performance. D&A is evaluated as a source of feedback between attempts for an open-loop CaP planner.
At every attempt, the planner observes the initial scene state and generates a code plan, which is then executed open-loop in the environment. Across attempts, the planner keeps track of previous plans and D&A feedback, which identifies failure modes and proposes corrections.
Acknowledgments
The authors thank Marc Toussaint and the Learning and Intelligent Systems lab at TU Berlin for the initial discussions that led to this project. Part of this work was supported by TRI U3.0, Ai2, and the Cross-Pacific AI Initiative (X-PAI), supported by Amazon, NVIDIA, the University of Washington, and the University of Tsukuba.
Frequently Asked Questions
What does D&A provide for downstream policy improvement? D&A provides dense progress rewards and rich language feedback.
Which policy-improvement paradigms are discussed here? The paper section outlines online and offline reinforcement learning and Code-as-Policies.
How is D&A used with Code-as-Policies? It provides feedback between attempts to an open-loop planner, identifying failure modes and proposing corrections.
What does the planner do at each attempt? It observes the initial scene state, generates a code plan, and executes the plan open-loop.