Researchers built OctoSense, an open-source platform and dataset combining stereo RGB, event cameras, LiDAR, thermal, IMU, RTK-GPS, and robot proprioception — then trained a single self-supervised model that fuses all these sensors into a unified representation. The result: a perception system that runs 6.68 milliseconds on an NVIDIA 5090 and outperforms image-only foundation models on optical flow, depth, semantic segmentation, and ego-motion estimation, especially in nighttime or sensor-degraded conditions.
What the Researchers Built
OctoSense is both a hardware sensor suite and a 59-hour dataset recorded from driving a car and a quadruped robot across diverse environments, times of day, and intentionally degraded sensor conditions (e.g., nighttime, dust, rain). The sensor suite includes stereo RGB cameras, stereo event cameras (which capture pixel-level brightness changes at microsecond resolution), a 32-beam LiDAR, a thermal camera, an inertial measurement unit (IMU), a real-time kinematic global positioning system (RTK-GPS), and proprioception data from the vehicle’s CAN bus and the robot’s joint angles.
On top of this physical platform, the team developed a self-supervised learning method called a late-fusion masked autoencoder. Instead of combining raw sensor data early (early fusion) or processing each sensor independently and averaging outputs (late fusion without interaction), their architecture uses separate tokenizers for each sensor modality to handle different spatiotemporal characteristics, then learns cross-modal representations by randomly masking tokens and reconstructing them. At inference time, the system caches previously computed modality-specific tokens so it can integrate new measurements as they arrive without reprocessing the entire history.
The trained model produces a single embedding — a high-dimensional vector — that encodes all sensor information. This representation can then be used for downstream tasks like depth estimation, optical flow, semantic segmentation, and ego-motion prediction without task-specific fine-tuning.
Key Results
The OctoSense model achieves inference times of 6.68 ms on an NVIDIA RTX 5090 GPU and 112 ms on an NVIDIA Jetson Orin NX, making it suitable for real-time robotics. The researchers compared their multimodal representation against image-only foundation models (e.g., DINOv2, CLIP, MAE) on four tasks:
- Optical flow: 12% lower error than the best image-only model.
- Depth estimation: 8% lower absolute relative error.
- Semantic segmentation: 5% higher mean intersection-over-union.
- Ego-motion (translation, rotation, steering angle): 15–20% lower error across all metrics.
The most striking advantage appears in degraded conditions: at nighttime, the thermal camera and event cameras provide complementary information, and the model maintains 90% of its daytime performance while image-only models drop by over 40%. In scenarios with simulated sensor dropouts (e.g., LiDAR failure), the model still produces usable representations by relying on remaining modalities.
No quantitative benchmarks were published for the quadruped robot portion of the dataset, but the driving test results demonstrate the core contribution — that a single self-supervised model can fuse many sensor types effectively.
How It Works
The OctoSense architecture is a late-fusion masked autoencoder designed to handle sensors with different frequencies, latencies, representations, and noise levels. The training process works as follows:
- Modality-specific tokenizers: Each sensor type (RGB, event, LiDAR, thermal, IMU, GPS, proprioception) first passes through a small neural network that converts its raw data into a sequence of tokens — compact vectors that encode the local spatial or temporal structure. These tokenizers are learned jointly with the rest of the model.
- Random masking: A large fraction (typically 75–90%) of all tokens from all modalities is randomly masked. The model must predict the masked tokens based on the unmasked ones, forcing it to learn cross-modal correlations.
- Transformer encoder: The unmasked tokens are fed into a Vision Transformer-style encoder that processes them together. Because the model sees tokens from different sensors in the same sequence, it learns relationships like “when RGB is dark, thermal spikes and LiDAR points become more informative.”
- Decoder: A lightweight transformer decoder reconstructs the masked tokens from the encoder’s output.
At inference time, the model does not need to reconstruct — it simply runs the encoder on any available sensor data to produce the embedding. A key practical trick: the system caches tokens from all sensors for the previous time step. When a new sensor measurement arrives (say, a LiDAR scan at 10 Hz while RGB runs at 30 Hz), the cached tokens for the slower modalities are reused while the new measurement is tokenized and combined. This avoids reprocessing the entire history.
The entire model is trained on the OctoSense dataset (59 hours) with self-supervision — no human annotations required beyond the automated time-synchronization of the sensors.
Why This Matters for Robotics
Most real-world robots — from autonomous cars to legged robots — operate in environments where any single sensor can fail or degrade. OctoSense demonstrates that self-supervised learning can fuse many sensors into one robust representation without expensive labeling. This is critical for:
- Autonomous driving: Nighttime and adverse weather reduce RGB camera performance, but thermal and LiDAR can compensate. A single model that handles all sensors simplifies the perception stack.
- Quadruped navigation: Legged robots experience vibration and variable terrain angles that degrade IMU and GPS; event cameras and thermal vision help maintain state estimation.
- Deployment on edge hardware: The 112 ms inference on Jetson Orin NX means the approach can run on embedded systems used in many field robots.
For operations managers and engineers evaluating robot platforms, the OctoSense dataset and code (publicly available) provide a ready-to-use baseline for multimodal perception. The ability to build a custom sensor suite and apply the same self-supervised method could reduce development time for inspection, autonomous navigation, and situational awareness tasks.
If you are integrating perception into an existing robot, consider browsing used industrial robots on Robot Overflow for platforms that can be retrofitted with multimodal sensors, or warehouse robots that might benefit from robust all-weather perception.
Limitations and Open Questions
The current OctoSense model was trained only on driving and quadruped walking data from a single city-dominant environment. Generalization to completely different domains — for example, underwater robots or drones — remains unverified. The tokenizers are learned per modality but are not flexible to adding new sensor types without retraining. The dataset is large (59 hours) but not as diverse as internet-scale vision datasets, which may limit representation quality for rare scenes.
Open questions include: how well does the model transfer to robots with different sensor placements or calibrations? Can the caching strategy handle sensors with highly irregular timing (e.g., event streams at microsecond granularity)? And does the late-fusion representation contain enough spatial detail for tasks like pixel-level manipulation, which may require early fusion?
Frequently Asked Questions
What sensors does OctoSense support? OctoSense includes stereo RGB cameras, stereo event cameras, LiDAR, a thermal camera, IMU, RTK-corrected GPS, and proprioception from CAN bus or joint angles — eight sensor modalities total.
Do I need to buy all these sensors to use the model? No. The trained model can work with any subset of those sensors at inference. The training dataset included all eight, but the architecture gracefully handles missing modalities via its caching and masking strategy.
How fast is the OctoSense model on edge hardware? It runs in 112 ms on an NVIDIA Jetson Orin NX, which is fast enough for real-time robotics (under 10 Hz for most perception loops).
Is the OctoSense dataset and code publicly available? Yes. The project page provides links to the dataset, code, and supplementary videos for replicating the results.
Conclusion
OctoSense proves that a self-supervised late-fusion masked autoencoder can combine eight diverse sensors into a single, robust representation that outperforms image-only foundation models on core perception tasks — especially in challenging conditions. With publicly available data and real-time performance on edge hardware, this work offers a practical blueprint for building multimodal perception systems that don't break when the sun goes down.