A team of researchers has released Deform360, the largest real-world visuotactile dataset ever built for deformable object manipulation. The dataset includes 198 everyday objects—from ropes and fabrics to plush toys—captured by 41 synchronized cameras and two tactile-equipped grippers across 1,980 interaction sequences, totaling 23.3 million frames. This massive resource gives roboticists the high-fidelity ground truth needed to train world models that can predict how soft, squishy, and flexible objects behave under manipulation.
What the Researchers Built
Deform360 is a curated collection of 1,980 physical interactions with 198 distinct deformable objects, recorded in a controlled multi-camera studio. Each interaction sequence involves a two-finger parallel-jaw gripper (the UMI gripper) equipped with tactile pressure sensors, performing a series of predefined actions—press, squeeze, twist, pull, drag—on the object while 41 calibrated cameras capture the scene from all angles.
The object taxonomy covers three major categories of deformability:
- 1D deformables (28 objects): ropes, cables, wires, belts, and chains with varying stiffness.
- 2D deformables (98 objects): fabrics, cloths, garments, bags, paper-like materials, and thin shells.
- 3D volumetric deformables (72 objects): plush toys, stuffed animals, foam objects, sponges, and squeezable household items.
For every episode, the researchers provide high-quality 3D reconstructions and particle tracking derived from a markerless multi-view perception pipeline. The tactile data captures normal-axis pressure at the gripper fingers, giving a direct reading of contact forces during each manipulation. This dense sensor fusion makes Deform360 a unique resource for studying how objects deform under real-world forces—not just visually, but physically.
Key Results
Deform360 represents an order-of-magnitude increase in scale over existing real-world deformable object benchmarks. With 23.3 million frames across 215.7 hours of footage, it dwarfs prior datasets that typically cover fewer than 20 objects in synthetic or single-view settings.
The researchers systematically evaluated two families of world models on this dataset:
- Video world models (Cosmos Predict 2.5, a diffusion transformer) fine-tuned on Deform360 with 7D action conditioning via cross-attention.
- 3D particle dynamics models that learn physics directly on the particle-based object representations.
Key findings:
| Model Type | Strengths | Weaknesses |
|---|---|---|
| Video world models (Cosmos) | Captures visual appearance and texture changes; leverages large-scale pre-training | Sensitive to out-of-distribution environments; reward design is difficult for planning |
| 3D particle dynamics | Accurate geometric prediction; straightforward to use with geometric cost functions (e.g., Chamfer distance) | Limited to geometric state; misses visual appearance; requires dense track data |
The evaluation revealed a fundamental trade-off: video models excel at visual novelty but struggle with precise geometry, while particle models offer better structural priors but lack visual richness. Notably, 3D particle models were successfully used for real-world model-predictive control (MPC) on deformable objects, whereas the video models required further post-training to handle visual domain shifts.
How It Works
The Deform360 capture system consists of 41 industrial cameras arranged in a hemispherical dome around the manipulation workspace. All cameras are calibrated and synchronized to record at 30 fps. The UMI gripper mounts two tactile sensors (GelSight-like) on its fingers, providing a 16×16 pressure map at each fingertip.
Object manipulation pipeline:
- Each object is placed on a matte black table in a fixed starting pose.
- The gripper executes a sequence of 5–10 predefined actions (press, drag, twist, pinch, push) lasting about 30 seconds per episode.
- All 41 camera feeds, tactile data, and robot joint angles are logged synchronously.
- A multi-view 3D reconstruction pipeline (based on NeRF and particle tracking) produces dense point clouds and particle trajectories for each frame.
World model training:
- Video models: The Cosmos Predict 2.5 architecture is a diffusion transformer (DiT) that generates future frames conditioned on past frames and future robot actions. The 7D action vector (6D wrist pose + gripper openness) is injected into the DiT blocks via cross-attention. Post-training on Deform360 adapts the model to the specific visual domain and action space.
- Particle models: Each object is represented as a set of 3D particles (typically 1,000–10,000). A graph neural network or neural ODE predicts particle motion given the current state and applied forces. Training uses the ground-truth particle trajectories from the reconstruction pipeline.
Why This Matters for Robotics
Robots in warehouses, factories, and homes must handle deformable objects daily: folding laundry, packing produce, assembling cables, or handling soft packaging. Yet most robotic systems rely on rigid-body assumptions that fail on deformable materials. Deform360 fills a critical gap by providing the high-fidelity, multi-modal data needed to train world models that can actually predict the behavior of ropes, fabrics, and plush toys.
For companies buying used cobots for sale or warehouse robots, progress in deformable object manipulation means fewer failures in tasks like bin picking, kitting, and sorting. In humanoid robotics, the ability to manipulate soft objects is essential for tasks like dressing assistance or object handover. The browse humanoid robots on Robot Overflow filter shows dozens of platforms that could benefit from these advances.
The preliminary real-world MPC demonstration—where a 3D particle model plans gripper actions to deform a rope into a target shape—proves that these world models can be deployed on actual hardware, not just in simulation.
Limitations and Open Questions
Heavy self-occlusion remains a challenge: when an object is largely hidden from most cameras (e.g., a rope coiled under a gripper), tracking quality degrades. The tactile sensors only measure normal-axis pressure, so micro-slip and tangential forces are invisible, and the no-slip regularization can overconstrain particles near contact points. The dataset is limited to two-finger gripper interactions, so behaviors requiring suction or multifingered grasping are not covered.
The video world models also face deployment hurdles: domain shift between the capture studio and real robot environments requires extensive post-training, and designing reward functions on raw video frames is significantly harder than on geometric state representations.
Frequently Asked Questions
What types of objects are included in Deform360? The dataset covers 198 objects divided into three categories: 1D deformables (ropes, cables, wires), 2D deformables (fabrics, cloths, bags), and 3D volumetric deformables (plush toys, foam objects, squeezable items).
How big is the Deform360 dataset? It contains 1,980 interaction sequences totaling 23.3 million frames and 215.7 hours of synchronized multi-view video and tactile data—an order of magnitude larger than any previous real-world deformable object dataset.
What sensors were used to capture the data? 41 calibrated synchronized cameras capture visual data from all angles, while two tactile sensors on the robot gripper measure normal-axis pressure at 16×16 resolution each.
What are the main challenges addressed by this dataset? Deform360 tackles the scarcity of high-fidelity, real-world data for deformable object modeling, enabling systematic evaluation of 2D video world models versus 3D particle dynamics models and providing a benchmark for future research.
Conclusion
Deform360 sets a new standard for deformable object datasets by combining unprecedented scale, multi-view visual coverage, and tactile sensing. The accompanying benchmarks reveal clear trade-offs between video-based and particle-based world models, giving the robotics community concrete guidance for choosing approaches based on task requirements. This resource will accelerate progress toward robots that can reliably handle the soft, squishy, and flexible objects that make up much of our physical world.