We present DenseReward, a dense vision-language reward model for robotic manipulation. The method consists of three components: an automated data generation pipeline that produces trajectories with phase-aware dense reward labels; failure synthesis that creates diverse failure trajectories through targeted perturbations; and DenseReward models trained on the resulting mixture of successful and failure trajectories to estimate fine-grained task progress.
Method
We present DenseReward, a vision-language reward model that predicts dense rewards for robotic manipulation. Our method consists of three components: an automated data generation pipeline that generates trajectories with phase-aware dense reward labels, failure synthesis that creates diverse failure trajectories through targeted perturbations, and DenseReward models trained on the resulting mixture of successful and failure trajectories to estimate fine-grained task progress.
Failure Synthesis
Building on the automated data generation pipeline, we synthesize failure trajectories with dense rewards.
DenseReward Training
Dataset. Based on our automated data generation pipeline, we construct a dense reward dataset containing 27k episodes from both successful and failure trajectories. The dataset covers diverse simulated and real-world sources, including: real-world success and failure episodes from DROID, simulated manipulation trajectories from RoboSuite, and simulated manipulation trajectories from Isaac Sim. The dataset covers diverse manipulation settings that include over 60 distinct manipulation objects. We split the dataset into training and test sets for model training and evaluation.
Conclusion
We present DenseReward, a dense vision-language reward model for robotic manipulation. We develop an automated simulation pipeline that generates trajectories with phase-aware dense rewards, and failure synthesis through targeted perturbations. DenseReward captures fine-grained task progress, partial completion, and common failure patterns. DenseReward outperforms general-purpose VLMs and existing robotic reward models in dense reward prediction, and provides useful reward guidance for downstream reinforcement learning. We hope our dataset, models, and evaluation suite will support future research on scalable reward learning for robotic manipulation.
Limitations and Future Work. We plan to extend this work to more complex manipulation, such as tool use and long-horizon tasks. Future directions also include incorporating human preference feedback into the reward learning process, to make dense reward models more general, scalable, and aligned with human expectations for real-world robot learning.
Frequently Asked Questions
How does DenseReward generate training data for dense rewards? It uses an automated simulation pipeline that produces trajectories with phase-aware dense reward labels, covering both successful and failure cases across over 60 distinct manipulation objects.
What types of failure trajectories does DenseReward use? Failure trajectories are synthesized through targeted perturbations in the simulation, creating diverse failure patterns that the reward model learns to recognize.
What sources provide the 27,000 training episodes? The dataset includes real-world episodes from DROID, plus simulated trajectories from RoboSuite and Isaac Sim, covering successful and failure cases.
How well does DenseReward perform compared to other methods? DenseReward outperforms general-purpose vision-language models and existing robotic reward models in dense reward prediction, and provides useful reward guidance for downstream reinforcement learning.