Autonomous vehicles and mobile robots rely extensively on LiDAR sensors to capture 3D point cloud representations of their surroundings. These sensors provide accurate geometric information that enables perception systems to understand complex outdoor environments. Within this perception stack, LiDAR semantic segmentation plays a central role by assigning a semantic label to every point in the 3D point cloud, allowing the system to distinguish between relevant scene elements such as vehicles, pedestrians, roads, and buildings. This semantic understanding is essential for downstream tasks including object detection, scene interpretation, and decision-making, which are critical for safe autonomous navigation.
Introduction
A unified evaluation protocol is structured around three complementary pillars that capture key requirements for autonomous driving perception.
Coarse-label evaluation assesses semantic segmentation performance under label groupings aligned with autonomous driving safety priorities. Beyond reporting results with coarser taxonomies, model performance is analyzed with fine- and coarse-grained labels, providing insight into the extent to which benchmark outcomes are influenced by annotation granularity. The COLA framework defines a safety-oriented taxonomy that groups semantically related classes according to their relevance for autonomous driving.
Robustness evaluation measures model performance under eight types of synthetic LiDAR corruptions designed to emulate realistic sensing degradations. These include atmospheric effects such as fog, rain, and snow; geometric perturbations such as motion blur and missing beams; and sensor-induced artifacts including crosstalk, incomplete echoes, and cross-sensor variations. Corruptions are analyzed both individually and by category, enabling systematic comparison of robustness characteristics across architectures. Inference speed on Jetson AGX Orin embedded hardware is reported alongside robustness results to quantify the trade-offs between robustness and computational efficiency.
Domain generalization evaluation assesses the ability of models trained on one dataset to operate in a previously unseen target domain without any adaptation. Specifically, models are trained on SemanticKITTI and nuScenes and evaluated directly on ParisLuco3D without any adaptation. This setting measures generalization across different geographic environments, acquisition conditions, and dataset distributions. By comparing performance on the source and target domains, the extent of cross-domain degradation is quantified and the robustness of the methods to distribution shift is assessed.
Using this protocol, a representative set of state-of-the-art methods spanning point-based, projection-based, sparse convolution-based, and fusion-based architectures is evaluated. All models are retrained from scratch under consistent training settings to ensure fair comparison. The resulting analysis provides a comprehensive characterization of the strengths and limitations of current LiDAR semantic segmentation approaches and reveals the trade-offs between semantic accuracy, robustness, generalization capability, and computational efficiency. These insights contribute toward a more realistic assessment of model suitability for autonomous driving applications.
Evaluation Protocols for LiDAR Semantic Segmentation
Evaluation of LiDAR semantic segmentation has traditionally followed the standard dataset partitioning of individual benchmarks, with SemanticKITTI and nuScenes serving as the primary references. These protocols measure in-distribution accuracy using fine-grained class taxonomies and clean point cloud data, which does not fully reflect the challenges of real-world deployment.
Unlike prior evaluation protocols, which typically focus on a single aspect of model performance, this protocol integrates semantic consistency, robustness, domain generalization, and computational efficiency into a unified assessment framework. By evaluating these dimensions under consistent experimental conditions, it provides a more comprehensive assessment of model suitability for real-world autonomous driving deployment.
Evaluation Protocol
A structured evaluation protocol assesses the deployment readiness of LiDAR semantic segmentation models. The protocol is organized around three complementary dimensions that characterize model behavior under conditions relevant to real-world autonomous driving: safety-oriented label semantics, adverse sensing conditions, and cross-domain transfer. In addition, computational efficiency is assessed through inference speed on both an NVIDIA RTX 4090 GPU and a Jetson AGX Orin embedded platform. All evaluated models are retrained from scratch under consistent training conditions to ensure a fair comparison.
Coarse-Label Evaluation for Safety Assessment
Standard benchmarks such as nuScenes and SemanticKITTI evaluate models using fine-grained class taxonomies with 16 and 19 classes, respectively.
While these annotations enable detailed scene understanding, fine-grained labels can lead to overly strict evaluations that penalize semantically reasonable predictions from a safety and operational perspective. In particular, confusions between functionally similar classes, such as car versus truck or bicyclist versus motorcyclist, may substantially reduce reported mean intersection over union despite having limited impact on many autonomous driving decisions. For example, confusing a car with a truck would often result in similar avoidance or braking behavior, even though it is counted as a complete error under conventional fine-grained evaluation.
To address this limitation, the COarse-LAbel framework groups dataset-specific semantic labels according to their functional role in autonomous navigation. Beyond reporting coarse-label performance, model performance is analyzed as it transitions from fine- to coarse-grained taxonomies.
Robustness Under Real-World LiDAR Corruptions
In real-world deployments, LiDAR sensors operate under a wide range of conditions that degrade point cloud quality in ways clean benchmarks do not capture. To evaluate robustness systematically, the corrupted datasets nuScenes-C and SemanticKITTI-C from the Robo3D benchmark are adopted. Models are trained on clean datasets and evaluated directly on corrupted versions without fine-tuning, ensuring the evaluation reflects out-of-distribution robustness.
Robo3D introduces eight corruption types organized into three semantically meaningful groups:
- Atmospheric: Fog, Wet Ground, and Snow, which are environmental conditions affecting laser propagation.
- Geometric: Motion Blur and Beam Missing, which represent motion dynamics and beam dropout effects.
- Sensor: LiDAR Crosstalk, Incomplete Echo, and Cross-Sensor, which reflect internal sensor characteristics or hardware differences.
The mean Corruption Error, or mCE, is adopted as the primary robustness metric.
Domain Generalization
Domain generalization evaluates a model’s ability to operate in previously unseen environments without any adaptation or access to target-domain data during training. In the context of LiDAR semantic segmentation, this requires robustness to shifts in sensor characteristics, environmental appearance, and scene composition.
Models are evaluated under a single-source-to-single-target protocol: a model trained on a source dataset is evaluated directly on an unseen target dataset without adaptation. nuScenes and SemanticKITTI are used as source datasets, and ParisLuco3D is used as the target dataset. ParisLuco3D was specifically designed for cross-dataset evaluation and differs from both training datasets in geographic environment, acquisition conditions, and scene composition.
Evaluation is performed using the native fine-grained label taxonomy of each source dataset. Since no label grouping or adaptation is applied, the resulting per-class intersection-over-union scores reflect the full impact of domain shift on detailed semantic segmentation performance.
Conclusion
A unified evaluation protocol was presented for assessing the deployment readiness of LiDAR semantic segmentation models through three complementary dimensions: coarse-label safety assessment, robustness to real-world corruptions, and cross-domain generalization, alongside inference runtime measurements on an embedded Jetson AGX Orin and an RTX 4090 GPU.
The results show that clean-benchmark accuracy is a poor predictor of deployment readiness. Methods that rank highly on standard metrics can fail on safety-critical classes such as living beings, degrade substantially under sensor corruptions, or suffer significant performance drops under domain shifts, with projection-based methods generally exhibiting greater sensitivity to sensor and scene variations. No single state-of-the-art method achieves a strong trade-off between accuracy, robustness, generalization, and computational efficiency.
The proposed protocol provides a reproducible framework for evaluating LiDAR semantic segmentation models beyond conventional benchmarks, enabling a more comprehensive assessment of deployment readiness, promoting more realistic evaluation practices, and guiding future model design for real-world autonomous driving perception.
Frequently Asked Questions
What does LiDAR semantic segmentation do? It assigns a semantic label to every point in a 3D LiDAR point cloud, distinguishing elements such as vehicles, pedestrians, roads, and buildings.
What are the three main evaluation dimensions? They are coarse-label safety assessment, robustness to LiDAR corruptions, and domain generalization across datasets.
Which datasets are used for domain generalization? Models are trained on nuScenes or SemanticKITTI and evaluated without adaptation on ParisLuco3D.
Which hardware platforms are used to measure inference speed? Inference speed is measured on an NVIDIA RTX 4090 GPU and a Jetson AGX Orin embedded platform.