RayViT injects per-pixel camera-ray geometry into pretrained vision transformers, giving imitation-learned robot policies a grounded 3D understanding of the scene. On a Franka Panda arm, the system keeps four long-horizon manipulation skills reliable even when external cameras are perturbed between trials. This directly targets one of the most common failure points for learning-based robots leaving the lab.
What the Researchers Built
RayViT is a visual representation method for imitation learning, designed to stop robot policies from breaking when the camera viewpoint changes. The researchers took off-the-shelf, pretrained vision transformers (ViTs) — the same architecture family behind modern vision-language models — and upgraded them with camera geometry. Every pixel in an image corresponds to a ray travelling through space, defined by the camera's intrinsics and its pose. RayViT injects that ray information into the transformer's patch tokens, so each visual feature knows not just what it sees, but where in 3D space it is looking. This is a deliberate departure from the common practice of treating multi-camera data as merely more pixels.
The training pipeline is multi-view. Observations from several cameras pass through the ray-conditioned encoder, and a cross-view consistency objective pulls their features into alignment with a chosen anchor view — the view most relevant to the task at hand. The resulting representation feeds a policy that outputs 8-dimensional actions: seven Franka joint positions plus gripper state. The team evaluated the system on a 7-degree-of-freedom Franka Panda across four real-world long-horizon manipulation tasks, with camera perturbations in the loop and a staged scoring metric that credits partial progress.
Key Results
The evaluation is built around a reality of deployed robots: cameras get bumped, remounted, and repositioned. RayViT's real-world tests include camera perturbations, and each long-horizon task is split into discrete stages, with the final score computed as the total number of successfully completed stages. That staged scoring matters — a robot that completes half a cup-stacking sequence earns partial credit instead of a flat zero. The setup mirrors production lines, where one wrong grasp can end a multi-step task and where partial progress is still meaningful information.
Four tasks make up the benchmark:
| Task | Goal | What varies |
|---|---|---|
| Cup Stack | Stack the orange cup onto the red, then both onto the blue | Cup vertical positions; horizontal positions fixed |
| Pick Place | Move a toy pear and sponge into a tray | Positions of the objects and tray contents |
| Drawer | Open upper drawer, insert a green prism, close drawer | Prism position; drawer fixed |
| Insertion | Drop a blue cylinder through a circular hole in a box | Cylinder position; box fixed |
Observations come from ZED Mini stereo cameras as external viewpoints and an Intel RealSense D405 as the gripper camera, with left-eye stereo images resized to 256×256. Actions are 8-dimensional across all tasks. The work's central demonstration is that a ray-conditioned, cross-view-consistent encoder keeps long-horizon manipulation policies functional when the camera setup changes — precisely the situation where viewpoint-memorizing policies typically degrade.
How It Works
At the heart of RayViT is a basic geometric fact: a camera image is a 2D projection of 3D rays. Each pixel sits on a ray originating from the camera center, fully determined by the camera's intrinsics (focal length, principal point) and extrinsics (position and orientation). Standard vision encoders ignore this and treat pixels as a flat 2D grid, which is why learned features end up tied to a single viewpoint.
RayViT changes this at the input level. For every patch token fed into the pretrained ViT, the method appends the parameters of the corresponding camera ray. The transformer learns visual features jointly conditioned on appearance and direction in space. When the camera moves, the ray parameters change accordingly, so the geometry-aware features map the same physical scene to a consistent representation rather than a different, alien image.
The second ingredient is multi-view training with a cross-view consistency objective. A task-relevant anchor view is selected, and features from all other viewpoints are trained to align with it. This teaches the encoder that the same manipulation event seen from different angles is the same event — an understanding no single-view dataset can provide.
The final policy consumes the aligned features and outputs an 8-dimensional action: seven joint positions plus gripper state. The paper's training details describe a diffusion-style noise schedule, with a starting noise level, a noise floor, and target-data standard deviation — the standard toolkit for today's diffusion policies. The result is a policy that reasons about the scene in 3D terms — where the cup is, where the gripper is, and whether the camera moved — rather than matching memorized 2D patterns.
Why This Matters for Robotics
Viewpoint brittleness is one of the biggest hidden costs in commercial robot deployment. A cell gets commissioned, a policy is trained with cameras in one position, and then a mount gets bumped, a camera gets replaced, or the rig moves to a new line — and success rates collapse. RayViT attacks that problem at the representation level, so the fix is portable: the same ray-conditioning idea can ride on any ViT-based control stack.
For buyers and integrators, the implications are direct. Robots that tolerate camera changes need less retraining, fewer re-commissioning cycles, and more freedom in how cells are laid out. That matters for used cobots for sale being repurposed on new lines, and for used industrial robots where camera rigs are rarely identical between installations. If imitation learning stops being viewpoint-fragile, it becomes a more credible tool for small-batch automation — and for humanoid robots operating in environments where no two camera views are the same.
Limitations and Open Questions
The authors are explicit about the boundaries of the approach. Only ViT-based encoders are considered; convolutional backbones, which still power a large share of robot vision, are not covered. The method is also developed in a multi-view setting and depends on a task-relevant anchor view for the cross-view consistency objective, which may require per-task judgment. Open questions follow naturally: how does the method behave with sparse or dense viewpoint sets, and can geometry-conditioned encoders be pretrained from the outset rather than retrofitted? The team flags that last direction — geometry-conditioned pretraining — as a promising path. If it succeeds, geometric awareness could become a standard ingredient of robot foundation models.
Frequently Asked Questions
What is viewpoint-robust imitation learning? It is imitation learning that keeps working when the camera angle or position changes. Standard policies often memorize a single viewpoint; viewpoint-robust policies are built to generalize across them.
How does RayViT know where a camera is pointing? Every pixel corresponds to a 3D ray defined by the camera's intrinsics and pose, and RayViT injects those ray parameters into the transformer's tokens. The encoder learns features that are aware of both appearance and geometry.
Does RayViT need multiple cameras at deployment time? No — multiple cameras are used during training to build the cross-view consistency objective. At inference, the ray-conditioned representations are what keep the policy robust to whatever viewpoint the camera ends up in.
What hardware was used in the experiments? A 7-DOF Franka Panda robot with ZED Mini stereo cameras for external views and an Intel RealSense D405 as the gripper camera. All observations were resized to 256×256.
Conclusion
RayViT treats camera geometry as first-class information instead of an afterthought, and that shift makes imitation-learned manipulation more portable across real-world setups. For anyone deploying learning-based robots, viewpoint robustness is no longer an academic nicety — it is becoming an operational requirement.