Introduction
Deep learning offers an opportunity to learn terrain-dependent effects directly from driving experience. A growing body of work has explored supervised and self-supervised traversability estimation for ground robots, using proprioceptive signals such as acceleration to generate supervision for visual terrain models. More recently, pretrained visual representations and visual foundation models have demonstrated promising transfer across visually diverse environments, reducing the amount of task-specific data required to learn representations of traversability. These developments suggest that rover driving telemetry and proprioception could provide useful labels about terrain suitability in addition to conventional geometric hazard labels.
However, planetary robotics lacks a large-scale dataset that directly connects driving experience with the multimodal observations required for learning traversability. Existing datasets surrounding Mars have primarily focused on visual semantic or terrain classification. For example, AI4MARS provides hundreds of thousands of semantic segmentation labels over imagery collected by the Spirit, Opportunity, and Curiosity rovers. While these annotations provide valuable information about visual terrain appearance, they do not directly encode how a six-wheeled rover physically interacted with the terrain.
In this work, a large-scale dataset is introduced from approximately 500 sols of Perseverance surface mobility operations. The dataset provides grayscale navigation camera (NavCam) stereo observations with rover poses, inertial measurements, rocker-bogie configurations, and estimated slip and tilt values, providing a multimodal record of both what the rover observed and how it responded to the terrain. Importantly, the trajectories represented in the dataset are not artificially generated: they arise from a combination of human Rover Planner decisions that are executed in conjunction with the onboard ENav software. The resulting data therefore captures the distribution of terrain, vehicle states, and driving conditions encountered during actual planetary exploration.
Built upon this dataset, this work investigates whether off-world driving experience can be used to learn a visual representation of terrain traversability. Visual foundation models are leveraged to extract transferable features, and a traversability estimator is trained using supervision derived from the rover’s proprioceptive data. The learned representation is subsequently projected into a traversability map that is integrated into the existing stereo-based planner, ENav. Rather than replacing the geometric autonomy stack, this approach augments it with a learned assessment of terrain.
The resulting system is validated on an Earth-equivalent rover platform with weight and mobility characteristics matching Perseverance on Mars. This evaluation provides a physical test of whether information learned from Martian driving data can improve navigation in closed-loop scenarios.
The contributions are threefold:
- A 500-sol multimodal driving dataset is introduced from the Mars 2020 (M2020) Perseverance rover. It contains synchronized stereo imagery, poses, inertial measurements, rocker-bogie angles, and tilt and slip estimates.
- A contrastive learning framework is presented that leverages pretrained visual foundation models and rover driving experience to estimate terrain traversability from visual observations.
- Closed-loop integration of the learned traversability representation with the ENav planner is demonstrated on an Earth-equivalent rover.
Together, the dataset and proposed learning framework establish a connection between operational planetary rover data and modern deep learning-based navigation, enabling future research in autonomous planetary exploration and navigation.
Learning-Based Terrain Traversability
These approaches are particularly relevant to planetary robotics because manual labeling of terrain according to rover mobility is impractical. The Perseverance driving logs provide a natural source of such supervision: visual observations can be associated with the rover’s observed motion, inertial response, vehicle configuration, and perceived slip. This enables traversability to be defined in terms of the behavioral response of the actual rover rather than a generic semantic terrain category. This work extends experience-based traversability learning to real planetary driving data collected over hundreds of sols.
Visual Foundation Models for Navigation
This work builds on the emerging use of visual foundation models for navigation while addressing the visual domain gap between off-world environments. Rather than relying solely on pretrained representations, Perseverance’s driving experience is used to adapt visual features to the rover-specific notion of traversability. This combination of foundation-model representations with long-duration planetary driving data provides a mechanism for transferring the broad visual knowledge of Earth-based pretrained models while grounding the resulting predictions in ground-truth rover-terrain interactions.
Model Architecture
The deployed model can be understood in three components:
- A pretrained, frozen DINOv2 ViT-L/14 encoder.
- A learned 5-million-parameter transformer decoder that generates a slip-organized space.
- A two-layer multilayer perceptron (MLP) head responsible for quantile prediction.
Traversability Prediction
Because the dataset contains a skewed distribution of slip readings, a quantile head is learned on top of the transformer instead of using regression. The quantile head is represented as a two-layer MLP that maps individual pixel features to quantile values.
The quantile head predicts a range of possible slip values for each pixel rather than a single slip estimate.
The MLP is trained to minimize the summed pinball loss of unseen sols from the perspective of the transformer. During inference, the final traversability is determined by taking the worst case between the median slip value and the predictive uncertainty, represented by the distribution width.
Latent Visualization
The 64-dimensional features of held-out wheel-track pixels are projected into two dimensions and colored by ground-truth slip. Principal component analysis (PCA) preserves dominant global directions of variation, while t-distributed stochastic neighbor embedding (t-SNE) emphasizes local neighborhood structure. Both projections use the same samples.
The PCA projection exhibits a continuous color gradient, with high-slip samples concentrated on the left and low-slip samples on the right. The representation therefore does not separate the data into “safe” and “hazardous” clusters; instead, one of its dominant directions varies continuously with slip magnitude.
The t-SNE projection shows complementary local structure: neighboring points share similar color while preserving the gradient. High-slip samples form multiple local groups rather than one isolated cluster, suggesting that elevated slip corresponds to more than one visual terrain appearance. The coherence of the low-slip region on the right indicates a consistent representation of benign driven terrain.
Hardware Experiments
The proposed method transfers to a visually out-of-distribution environment on an Earth-analog testing site. Testing was performed primarily on flat ground, with a variety of large rocks serving as obstacles.
As the rover navigates around various boulders, the model generates a traversability map and projects it onto a wedge. The modified planner accepts the traversability scores and fuses them with the original geometric-based cost map to search for a path.
Conclusion
A dataset and learning framework are presented for dense, vision-only traversability prediction from Perseverance data. By projecting the rover’s wheel tracks into pre-traversal NavCam imagery, the method contrastively ranks visual features against proprioception and suspension response without requiring manual annotation.
A dense decoder is trained over features generated from a DINOv2 encoder using complementary objectives. A Rank-N-Contrast loss structures features within the driven region, while bidirectional cross-modal alignment relates those features to the rover’s mechanical response.
Future work could support online adaptation to new terrains or scale the method toward larger backbones such as a vision-language model.
Acknowledgements
This research was carried out at the Jet Propulsion Laboratory (JPL), California Institute of Technology, under a contract with the National Aeronautics and Space Administration. This work was supported by JPL’s Strategic Research and Technology Development (SRTD) program.
The High Performance Computing resources used in this investigation were provided by funding from the JPL Enterprise IT Division. The authors thank Tyler Del Sesto and Richard Rieber for their insightful discussions on the hardware platform and ENav algorithm, along with JPL’s Rover Operations Center (ROC) for its support.
Frequently Asked Questions
What dataset does this work introduce? It introduces a multimodal driving dataset collected over approximately 500 sols of Perseverance surface mobility operations.
What rover data supervises traversability learning? Supervision is derived from visual observations, rover motion, inertial response, vehicle configuration, perceived slip, and suspension response.
What visual model is used by the deployed system? The model uses a frozen DINOv2 ViT-L/14 encoder, a learned transformer decoder, and a two-layer MLP quantile head.
How is the learned traversability representation evaluated? It is integrated with the ENav planner and tested in closed-loop navigation on an Earth-equivalent rover.