We present WEAVER, a large-scale generative world model for robotic manipulation that enables efficient policy evaluation, improvement, and test-time planning through high-quality imagined rollouts. WEAVER outperforms prior approaches across multiple manipulation tasks, generating longer, more accurate predictions with faster inference.
Results
We first study the performance of the WEAVER world model in isolation and then in the downstream use-cases of policy evaluation, improvement, and test-time planning.
Conclusion
Broader Impact. This work explores large-scale world models to improve the efficiency, safety, and scalability of robotic manipulation by reducing reliance on costly real-world interaction. Imagined rollouts can support policy evaluation, improvement, and test-time planning before execution, but inaccurate or biased predictions may lead to risky decisions that are particularly important in safety-critical domains like assistive robots. Responsible deployment therefore requires careful validation, uncertainty estimation, and safeguards against exploiting errors in learned world or reward models.
Task Details
PnP Bag requires the robot to pick up a bag of chips and place it on a green plate. We use two types of chip bags and randomly sample one in each episode. The bag is deformable, making the grasp outcome and object motion difficult to predict.
PnP Marker requires the robot to pick up an Expo marker lying horizontally on the table and place it inside a container. The marker color is randomly selected from black and orange, and the target container is randomly selected from a paper cup and a blue mug. This task requires precise grasping and large end-effector reorientation to insert the marker vertically.
PnP Towel requires the robot to pick up a towel and place it into a basket. We use two towel variants, a folded thick red kitchen towel and a thin gray square towel, and two basket variants, orange and blue. The task is challenging because the towel is deformable, and its resulting shape depends strongly on the grasp location and, for the folded towel, the number of layers grasped.
Pour Beans requires the robot to pick up a cup containing coffee beans and pour them into a blue bowl. This task tests dynamic manipulation, as the granular motion of the beans is difficult to predict and successful execution requires accurate control of cup pose, pouring angle, and motion to avoid spilling outside the bowl.
Action Space
For policy evaluation, because the joint positions at the end of each trajectory are already available, we directly use joint positions as inputs to the world model. During test-time planning, however, predictions must be made from the joint-velocity actions proposed by the policy. We therefore use the trained action adapter to predict the corresponding joint positions, and condition the world model generation on these adapted actions.
The action adapter is a lightweight feedforward module that bridges the world model's action representation (joint velocity commands and binary gripper signals) and the robot's observable state (absolute joint positions and gripper width). Given the robot's current state and a chunk of up to 10 actions produced by the world model, it predicts the resulting sequence of joint-position and gripper-position deltas, which are then integrated to obtain future absolute states.
Partial Observability
Our world model relies primarily on visual observations, which provide only partial access to the underlying physical state. During manipulation, task-relevant information such as object contacts, grasp stability, applied forces, or occluded object geometry may be hidden from all available camera views. This limitation is especially pronounced for wrist-camera observations, where the viewpoint changes continuously, and for cluttered scenes where objects may leave the field of view or become occluded by the gripper. Although memory and multi-view conditioning mitigate this issue, purely visual prediction may still fail when the missing state cannot be inferred from image history alone. Incorporating additional sensing modalities, such as tactile feedback, force-torque sensing, or depth, may improve state estimation and long-horizon prediction under occlusion.
Complex Deformable and Dynamic Interactions
Deformable-object manipulation and dynamic manipulation remain challenging for learned world models. Objects such as towels, bags, and granular materials exhibit high-dimensional, history-dependent dynamics that are difficult to capture from limited robot data. Small errors in predicted contact, grasp location, or object configuration can compound over time and lead to qualitatively incorrect rollouts. This is particularly evident in tasks such as pouring, where the motion of granular material depends sensitively on cup pose, velocity, and contact with the container. Future work may improve prediction fidelity by incorporating physics priors, hybrid neural-physics models, or neural simulators specialized for deformable and granular dynamics.
Limited Planning Horizon at Test Time
Although our inference acceleration strategies make test-time planning feasible with a large generative world model, latency still limits online planning to a single action chunk. As a result, the planner can improve near-term action selection but cannot yet perform long-horizon lookahead. This restricts its ability to reason about delayed consequences or multi-stage recovery behaviors. Further improvements in sampling efficiency, model distillation, value estimation, or hierarchical planning could enable longer-horizon online reasoning while maintaining real-time control.
Data Coverage and Embodiment Diversity
Our world model is pretrained primarily on DROID, which provides large-scale robot interaction data but is still tied to a specific robot embodiment and data collection setup. This may limit generalization to substantially different robots, camera configurations, end-effectors. In addition, some task dynamics in our evaluation, such as granular pouring, are underrepresented in the pretraining data. Scaling world-model training to more diverse sources, including cross-embodiment robot datasets, simulation data, and human videos, may improve robustness and broaden the range of behaviors that can be accurately imagined.
Frequently Asked Questions
What makes WEAVER different from other world models for manipulation? WEAVER achieves longer and more accurate imagined rollouts than prior approaches while maintaining faster inference, making it practical for online test-time planning.
Which manipulation tasks were used to evaluate WEAVER? The evaluation includes pick-and-place of deformable objects (bags, towels, markers), and dynamic manipulation (pouring granular materials like coffee beans).
What are the main limitations of the current WEAVER model? Key limitations include partial observability from visual inputs, difficulty with complex deformable dynamics, limited planning horizon during test time, and restricted embodiment diversity in training data.
How does WEAVER handle the gap between velocity actions and joint position inputs? A lightweight action adapter module predicts joint-position deltas from velocity commands, bridging the action representation gap for world model conditioning.