EyeRobot 2.0 is a robot learning framework that introduces Active Visual Fixation (AVF). Using input only from one fixed stereo camera, it can carry out precise manipulation tasks such as inserting a straw, capping a marker, zipping a bag, or extracting a wrench from a toolkit, as well as multi-stage tasks such as opening a toaster oven, pulling out the tray, extracting a pan, pushing the tray back, and closing the door.
We evaluate EyeRobot 2.0 on seven physical tasks and six simulated MuJoCo tasks, collecting data and training single-task policies from scratch. Across more than 1,000 physical trials, it outperforms static stereo baselines by 40%, and by 20% on simulated pick-and-place tasks, with a larger performance improvement on fine-precision tasks. Compared to the widely adopted state-of-the-art ego-plus-wrist setup, EyeRobot 2.0 without wrist cameras matches task performance in physical trials. On tasks with heavy occlusion of wrist cameras, it maintains strong performance while wrist-camera baselines regress to static stereo performance. Ablations show that fixation-centric actions account for a significant portion of EyeRobot 2.0’s improvement, providing a 20% performance boost across simulation tasks. Foveation contributes strongly to real-world success on fine-grained tasks, boosting performance by 20% overall.
Manipulation with Active Visual Fixation
We study visuomotor policy learning from human teleoperation demonstrations for bimanual manipulation with parallel-jaw grippers. Given a set of demonstrations specifying a task, the goal is to train task-specific policies that reproduce the demonstrated behavior with bounded variations in pose.
EyeRobot 2.0 considers an ego-centric setting in which observations are provided solely through an approximately head-mounted stereo camera. The policy is evaluated using task-specific success criteria.
Learning Gaze Patterns with Reinforcement Learning
We train AVF in two phases. First, we train a goal-conditioned fixation policy capable of fixating on any task-relevant objects. Next, we train a hierarchical target-selection policy that sequences fixations to optimize task performance.
In the first phase, the fixation policy is trained with reinforcement learning using a geometric reward from detected object positions, enabling accurate 3D tracking of prompted objects. In the second phase, the target selector is trained with reinforcement learning in a behavioral cloning and reinforcement learning loop: a gripper behavioral cloning policy is co-trained on the resulting fixated views, and the selector is rewarded for fixations that maximize the behavioral cloning policy’s action-prediction accuracy. This hierarchical formulation allows task-dependent gaze behavior to arise without manual human specification.
Experiments
EyeRobot 2.0 is implemented using a bimanual, 14-degree-of-freedom I2RT YAM manipulator with parallel-jaw grippers. We mount a static stereo camera between the arms, 35 cm above the base joint and angled 45° downwards for full observability of the table. Each eye captures a 120° diagonal field of view at a resolution not specified in the provided text. We use a custom leader arm, modified from GELLO and TRLC, to teleoperate the robot.
Results
We evaluate how well EyeRobot 2.0 performs compared to state-of-the-art baselines. To construct carefully controlled comparisons, we collect all data ourselves and train all policies from scratch on the exact same demonstrations. This avoids obfuscation from unknown pretraining distributions of off-the-shelf models, reduces compute demand, and increases the tractable volume of experiments by an order of magnitude.
Real-world evaluation
We train every policy with three seeds and randomly select one for evaluation. For each task, we evaluate all policies on 25 scene configurations selected by placing objects within the training-data region, but not exactly matching training datapoints. During evaluation, we overlay the live camera stream on the test positions and manually align objects so all policies share the exact same test positions. Across all real-world physical trials of baselines and ablations, we evaluate more than 1,000 policy rollouts.
Simulation evaluation
Every baseline and EyeRobot 2.0 policy is trained across three seeds; we report the maximum performance to prune failed seeds. All simulation evaluations are run on the same 100 random configurations, unseen during training but drawn from the same distribution. Each policy is trained from scratch on single-task data and executed using the same temporal-ensembling weights and inference code.
The gaze policy is given one second to allow gaze to settle before acting. Per-stage success is judged programmatically using manually specified MuJoCo physics requirements, and each task has a failure timeout based on the average length of demonstrations.
Ablations
We ablate the multi-scale image pyramid and train both gaze and gripper policies on one peripheral view for each eye. Overall, this harms performance by 20%, with the largest gap on precise physical tasks that require detailed image understanding.
In simulation, this ablation performs the same as EyeRobot 2.0. This may be because robot execution in simulation has little to no variance, whereas real-world execution varies from trial to trial and therefore demands more closed-loop visual servoing.
Limitations and Future Work
EyeRobot 2.0 currently trains only task-specific gaze and gripper policies. An interesting direction for future extension is to investigate how to adapt the framework to maintain multi-task performance. Using a target selector means gaze would need to be trained on a sufficiently large corpus to allow generic prompts, which may necessitate different architectures for generating prompts, such as a vision-language model.
This work does not control head or neck movement; it proposes how to coordinate fine-grained eye movements when a good head position is already available. Finally, while object-level pretraining is sufficient for many manipulation tasks, it cannot yet train sub-object-level gaze, which may provide additional benefits for manipulation.
Conclusion
EyeRobot 2.0 is a robot learning framework for fine-grained bimanual manipulation using only a fixed stereo camera. It leverages active visual fixation to direct attention towards task-relevant features. The experiments suggest that active fixation makes more effective use of stereo data than non-active stereo baselines, strongly outperforming them across a range of high-precision physical tasks and matching or exceeding the performance of wrist cameras.
Ablating any of its components results in a performance drop, indicating that EyeRobot 2.0 uses multi-resolution image inputs, fixation-centric actions, and stereo information to facilitate fine-grained real-world manipulation. This suggests potential for systems that use active visual fixation for fine-grained manipulation instead of wrist cameras.
Selected Hyperparameters
All training code will be released for reproducibility; selected important hyperparameters are reported here.
The gaze policy discretizes its action space and outputs a cross-product categorical distribution. Azimuth and elevation are binned into eight polar directions, with two magnitudes for each direction (10° and 3°), along with a zero action. Depth deltas are binned with three magnitudes (10 cm, 5 cm, and 1 cm) and a zero action. The resulting cross-product action space has 119 bins for the policy output. Discretized action spaces are preferred because they help break ties in ambiguous actions, such as duplicate objects.
Test Distribution
Policies are evaluated on the exact same positions, within human placement error, by aligning object positions with an overlay of the current position and test position.
Distractor Objects
For distractor experiments, we follow the same procedure as for full evaluations, recording 25 positions of test objects and distractor objects and exactly repeating them for all models.
Per-Stage Success Criteria for Real Tasks
For all stages, we allow three retries before counting a stage as a failure. We also count a failure if the robot stops moving for more than 10 seconds.
- Pick yellow: The robot must grasp the yellow tape and lift it fully off the table without dropping it before the next stage.
- Handover: The robot must successfully transfer the yellow tape to the right gripper without dropping it.
- Place: The robot must place the yellow tape on top of the grey tape without it falling off or tilting. The static position must be stable, with the center of gravity fully supported by the grey tape roll.
Frequently Asked Questions
What is Active Visual Fixation? It is a learned approach that directs a robot’s gaze towards task-relevant objects and features.
What camera setup does EyeRobot 2.0 use? It uses input solely from one fixed stereo camera mounted between the robot’s arms.
How does EyeRobot 2.0 perform compared with wrist-camera systems? It matches their task performance in physical trials and maintains strong performance in tasks where wrist cameras are heavily occluded.
What are the main limitations described? The framework currently trains task-specific policies, does not control head or neck movement, and cannot yet train sub-object-level gaze.