Astra, a vision-and-touch robot policy, completed contact-rich manipulation tasks in simulation, reaching a 93.8% mean on UniVTAC and at least 90% success on 13 of 19 tasks. The results show that tactile images can help a general multimodal model manage contact without a separate task-specific tactile policy, while also exposing where touch adds little, visual cues mislead, and corrective movements still fail.
What did the researchers build?
The researchers evaluated Astra, a robot policy that uses visual observations and camera-like tactile images to control contact-heavy manipulation. Rather than treating touch as a separate stream requiring a newly trained controller for every task, Astra processes tactile images alongside scene images, robot state, previous motion, and information from demonstrations. The tactile sensors represent contact through images of local surface deformation.
Astra generates ordered commands for the robot’s end effector—the part that interacts with objects—and its gripper. It then uses new observations to decide what to do next. That feedback matters because issuing a command is not the same as making progress: an insertion motion, for example, can leave an object caught against the rim.
The evaluation covers two simulated benchmarks, UniVTAC and OpenViTac, with tasks involving classification, object selection, grasping, regrasping, and insertion. Examples include identifying surface roughness, distinguishing objects by weight, and guiding parts toward constrained openings. The work tests how far a multimodal model can go with contact information, and where it still struggles to turn that information into reliable physical corrections.
What results did Astra achieve?
With tactile input, Astra reached at least 90% success on 13 of 19 tasks, including perfect success in the sampled trials for nine tasks. Its mean success on UniVTAC was 93.8%. But the value of touch varied by benchmark: adding tactile input improved results by 17.5 percentage points on UniVTAC and just 1.2 points on OpenViTac.
| Reported result | Finding |
|---|---|
| UniVTAC mean success with tactile input | 93.8% |
| Tasks reaching at least 90% success with tactile input | 13 of 19 |
| Tasks with perfect sampled success | 9 |
| Tactile-input gain on UniVTAC | 17.5 percentage points |
| Tactile-input gain on OpenViTac | 1.2 percentage points |
These are sampled simulation results, not guarantees of real-robot performance. Most tasks used only ten episodes per input setting, so one outcome can shift a task’s success rate by ten percentage points.
Touch also did not help every task. Results fell on Weight Classify, Empty-Can Selection, Insert Tube, and Insert Block v2 when tactile input was added. In three classification or selection tasks, Astra also used visible textures, ridges, or object contours to make decisions without tactile images. That suggests some apparent “touch” tasks can be solved through visual clues when appearance correlates with the answer.
How does Astra turn vision and touch into actions?
Astra combines observations from the environment with information about the robot and its recent behavior. The visual input can include fixed and wrist-mounted cameras. Tactile sensors provide images in which contact changes the visible surface pattern—for example, deformation where a fingertip presses against an object. The policy can compare those patterns with object appearance, robot state, preceding motion, and demonstrated outcomes.
From that context, Astra produces a sequence of end-effector and gripper commands. The commands move or grasp objects; new observations then let the policy check what actually changed. If an object remains blocked at an opening, the policy can revise its next action instead of treating the attempted insertion as a completed task. Brief action notes connect a decision to Astra’s assessment of the situation, helping researchers analyze whether the action matched the observed change.
This loop is especially important for contact because the same movement can produce different outcomes depending on alignment, support, and friction. A tactile image can show that contact changed, but interpreting the change is harder: a small deformation might indicate a light object, a particular grasp, or little motion between the object and fingers. Reliable control requires the policy to distinguish those possibilities and choose a useful next movement.
The results also show why vision and touch need to be interpreted together. Astra sometimes spotted visible surface features and completed a task without touch. In other cases, tactile input added useful evidence, but the policy still struggled to translate a diagnosis—such as detecting an obstruction—into a correction that made the object move as intended.
Why does visuo-tactile control matter for robotics?
Many useful robot jobs involve more than seeing an object and moving toward it. A robot may need to hold an item without dropping it, align a part with a tight opening, or respond when contact reveals that the first motion did not work. Astra’s results are relevant because they test whether a general multimodal policy can use image-based touch as part of that decision loop, rather than relying only on a task-specific tactile controller.
For robot buyers and operations teams, the results are a reason to ask how a system handles failed contact—not just whether it can complete a clean demonstration. A useful evaluation should show whether the robot notices a slip or obstruction, identifies a likely cause, and makes a correction that improves the situation. Those questions apply when comparing warehouse robots and humanoid robots, though this study does not establish performance on either type of physical platform.
The benchmark findings also matter for evaluation design. If a task’s visual appearance gives away the answer, a high score may reflect visual recognition rather than physical understanding. More revealing tests would vary appearance independently from weight, friction, stiffness, or other properties, and would include longer contact sequences, tighter tolerances, and recovery from failed grasps.
What are Astra’s limitations?
All experiments were conducted in simulation. Physical robots add sensor noise and drift, calibration error, real contact dynamics, and execution delays—conditions not validated here. The study also covers only two benchmarks and uses task-specific successful demonstrations. Tests on unfamiliar objects, tasks, sensors, and robot configurations remain open questions.
The sampled evaluation is small: most tasks use ten episodes per input setting, and results can change substantially with a single success or failure. Larger samples and repeated runs would give a clearer picture of performance variation.
The failure cases reveal practical control gaps. Astra sometimes misread weak tactile cues, struggled to make fine adjustments, or continued probing without reaching a decision in time. In insertion attempts, detecting that a part was still blocked did not ensure that the next pitch, yaw, or depth adjustment would solve the problem. The system can recognize a problem without reliably diagnosing its cause or producing an effective correction.
What are the frequently asked questions about Astra?
What is Astra? Astra is a vision-and-touch robot policy evaluated on contact-rich manipulation tasks in simulation. It produces gripper and end-effector commands from observations and feedback.
What does tactile input look like in this system? The tactile sensors produce images that show local deformation during contact. These images give the model visual patterns to interpret alongside camera views and robot state.
Does touch always improve Astra’s performance? No. Tactile input improved results by different amounts on the two benchmarks and reduced success on four reported tasks.
Has Astra been tested on physical robots? No. The reported experiments are in simulation, so robustness on real hardware remains unverified.
Astra shows that image-based touch can help a general multimodal policy handle a broad set of simulated manipulation tasks. Its remaining challenge is turning contact observations into efficient, reliable corrections—and proving that capability on physical robots.