RoboDream is a generalizable embodiment-centric world model for robot data synthesis. By conditioning video generation on explicit scene and object priors, our method enables controllable demonstration generation across arbitrary environments and objects without task-specific fine-tuning. Two practical deployment modes, retrieval and rebirth and prop-free teleoperation, address key bottlenecks in real-world data collection. Experiments show that RoboDream consistently improves policy performance and reduces the burden of real data collection.
Prior Extraction
To train RoboDream, we construct training pairs matching scene, object, and motion embeddings from existing large-scale robot datasets. This automated pipeline extracts priors without manual annotation.
Object Identification. Given the first frame and task instruction, we use a vision-language model (GPT-5-nano) to identify the objects explicitly relevant to the task. The model returns a list of object names (e.g., "red cup," "blue sponge"), filtering out background elements like tables or walls.
Scene Prior Construction. We generate the scene prior by removing the segmented task-relevant objects from the first frame. We utilize OmniPaint, a diffusion-based inpainting model, to fill in the holes left by object removal, resulting in a clean background image of the workspace.
Experimental Setup
We conduct experiments on a Franka Panda robot setup consistent with the DROID platform. Each task involves precise object manipulation. We evaluate success based on task completion; for pick-and-place tasks, we consider partial success (e.g., successful grasp but failed placement) as half credit. All real-world evaluations are conducted over 20 rollouts per policy.
Scaling Up With Generated Data
We further study the scaling properties of our data engine. We fix the real dataset at 50 demonstrations and progressively increase the number of generated demonstrations mixed in. The generated data consists of retrieved DROID trajectories reborn in our target setting. Table III reports success rates as generated data scales from 100 up to 400 additional episodes.
We observe that adding generated data consistently improves performance over the Real-50 baseline. The performance gains appear to saturate at approximately Mix-200. This suggests that while RoboDream provides a scalable source of data, the diversity of retrieved trajectories (from DROID) or the domain gap in generation eventually limits the marginal benefit of adding more samples. Nevertheless, the ability to significantly improve policy performance without collecting additional real data demonstrates the power of our data engine.
Compositional Generation
Finally, we investigate the model's zero-shot generalization capabilities through compositional generation. RoboDream effectively decouples the components of manipulation, allowing us to recombine them to generate diverse demonstrations in a zero-shot manner.
Limitations
RoboDream assumes that the rendered robot-only video faithfully captures the kinematic trajectory that the robot should follow in the target scene. While generation quality depends on the coverage of the training distribution, we believe this can be resolved by further scaling up training with extensive data, including human videos. By treating the human as an embodiment and grounding generation to human motion, our framework can potentially leverage internet-scale human data to improve robustness. Additionally, the model inherits the temporal length and resolution limitations of the underlying video diffusion backbone. We expect these limitations to diminish as base video generation models continue to improve.
Conclusion
We presented RoboDream, a generalizable embodiment-centric world model for robot data synthesis. By conditioning video generation on explicit scene and object priors, our method enables controllable demonstration generation across arbitrary environments and objects without task-specific fine-tuning. Two practical deployment modes, retrieval and rebirth and prop-free teleoperation, address key bottlenecks in real-world data collection. Experiments show that RoboDream consistently improves policy performance and reduces the burden of real data collection. We believe this work provides a practical path toward scalable imitation learning without the need for massive teleoperation efforts or task-specific simulation setups.
Acknowledgment
We are grateful to Jiawei Yang, Mingtong Zhang, and Weiduo Yuan for their helpful discussions, and to Abrar Anwar and Jesse Thomason for their support with the robot hardware. The USC Physical Superintelligence Lab acknowledges generous support from Toyota Research Institute, Dolby, Google DeepMind, Capital One, Nvidia, Bosch, NSF, and Qualcomm. Junjie Ye is supported by a Capital One Fellowship. Yue Wang is also supported by a Powell Research Award.
Frequently Asked Questions
What is RoboDream's core innovation? RoboDream is an embodiment-centric world model that conditions video generation on explicit scene and object priors, enabling controllable robot demonstration synthesis without task-specific fine-tuning.
How does RoboDream extract priors from existing data? It uses a vision-language model (GPT-5-nano) to identify task-relevant objects from the first frame, then employs OmniPaint to inpaint the background, generating a clean scene prior.
What experiments validated RoboDream's performance? The method was tested on four real-world manipulation tasks with a Franka Panda robot, measuring success rates across 20 rollouts per policy and showing consistent gains when mixing generated data with real demonstrations.
What are the main limitations of RoboDream? The model assumes rendered robot-only video faithfully captures the kinematic trajectory, depends on training distribution coverage, and inherits temporal length/resolution constraints from the video diffusion backbone.