Long-term place recognition in unstructured agricultural environments is challenging due to significant seasonal changes and high visual similarity between nearby locations. This work introduces a heterogeneous LiDAR fusion strategy that combines data from a Livox Mid-360 solid-state LiDAR and a Velodyne VLP-16 spinning LiDAR, leveraging the short-range strengths of Livox and long-range capabilities of Velodyne. Additionally, a learned re-ranking strategy refines the top candidates retrieved by a MinkUNeXt-based backbone, improving retrieval accuracy. Experiments on the TEMPO-VINE vineyard dataset show substantial improvements over state-of-the-art methods, with Recall@1 increasing from 16.4% to 34.3% in cross-season scenarios. A novel evaluation protocol based on phenological stages further demonstrates the method's robustness.
Introduction and Related Work
Although methods such as HeLiOS process a single point cloud independently using a shared encoder with spherical transformers, the literature has less explored the explicit fusion of data from different LiDAR sensors. There are no existing methods that specifically address the fusion of heterogeneous LiDAR data in a unified representation for place recognition in unstructured environments, which is the focus of this work.
Re-ranking strategies are particularly important in agricultural environments because the high similarity between places and crop occlusions in the latter stages can lead to a high number of false positives in the initial retrieval stage. Two places within the same trajectory that are in very different positions (intra-row and extreme) show little difference. This makes retrieval a non-trivial problem in these types of settings. By refining the ranking of candidates, re-ranking approaches can significantly improve the performance of LPR algorithms in these challenging environments.
Sensor Fusion Strategy
The main differences between two LiDAR sensors with different scanning patterns are considered when fusing data from them. The Livox Mid-360 is a solid-state LiDAR sensor that provides a wide field of view (FoV) of 360 degrees horizontally and 38.4 degrees vertically. Its non-repetitive scanning pattern allows it to capture more points in a given area than traditional spinning LiDARs. In contrast, the Velodyne VLP-16 is a spinning, mechanical LiDAR sensor that provides a narrower FoV of 360 degrees horizontally and 30 degrees vertically. It uses a repetitive scanning pattern, resulting in a more uniform distribution of points, though it may miss certain areas due to occlusions or limited resolution.
Livox provides reliable information at short ranges, while Velodyne performs better at long ranges. Therefore, a fusion strategy is proposed that leverages the strengths of each sensor. First, the point clouds from both sensors are downsampled to a common resolution. For coherence, the points of the Velodyne sensor are then transformed to the common frame between both sensors using the extrinsic transformation. A distance threshold of 10 meters is defined to distinguish between short- and long-range areas. Points from the Livox source are selected exclusively for the short-range area and from the Velodyne source exclusively for the long-range area. This method yields a fused point cloud that combines the strengths of both sensors without the need for complex approaches or high computational overhead. Finally, the selected points from both sensors are combined to create a fused point cloud for LPR.
Learned Re-Ranking Head
The re-ranking head is trained using a Binary Cross Entropy (BCE) loss, where the positive samples are the true matches and the negative samples are the false matches among the top K candidates. The BCE loss function computes the cross-entropy between the true labels (1 for positive matches, 0 for negative matches) and the predicted probabilities of being a positive match.
This additional head is trained in inference time for 25 epochs with a batch size of 1024. The optimizer used is Adam with a learning rate of 0.001. The re-ranking head is designed to be lightweight and efficient, ensuring that it does not introduce significant computational overhead during inference. For large evaluations involving at least three campaigns and their corresponding trajectories, each epoch takes around one minute on an Nvidia A30 GPU.
Evaluation Metrics and Dataset
For evaluating the performance of the method, Recall@1% and Recall@1 metrics are used. Recall@1% measures the percentage of queries for which at least one correct match is found within the top 1% of the retrieved candidates. Recall@1 measures the percentage of queries for which the top retrieved candidate is a correct match. These metrics are commonly used in place recognition tasks to evaluate the effectiveness of retrieval methods. A higher value in the Recall@1 metric is crucial, as it indicates that the top retrieved candidate is a correct match, which is essential for accurate localization in real-world applications.
Every experiment in this paper is performed using the TEMPO-VINE dataset, while results for the BLT dataset are also reported for the ablation study of the re-ranking contribution.
Cross-Season Evaluation Results
The performance of MinkUNeXt-VINE++ is evaluated in a cross-season scenario by grouping campaigns into three different phenological stages: early growth, flowering, and pre-harvest. The model is trained using data from one phenological stage and tested using data from a different phenological stage to demonstrate cross-season generalization capability.
Conclusions and Future Work
As future work, a more scalable fusion strategy is planned that can be applied to a wider range of sensors and environments. Additionally, the application of the method will be extended to other environments, such as urban settings, to further validate its effectiveness and generalization capabilities. Regarding the re-ranking head, the next objective is to create an extensive benchmark of the effects of different input descriptors and seasonal changes on correct reordering in agricultural environments. Finally, the goal is to deploy the method on an autonomous system operating in a real-world vineyard environment to demonstrate its practical applicability and robustness.
Frequently Asked Questions
What is the main contribution of this paper? The paper proposes a heterogeneous LiDAR early fusion strategy that combines Livox Mid-360 and Velodyne VLP-16 data based on range-specific strengths, along with a learned re-ranking head to refine retrieval candidates in agricultural environments.
How does the re-ranking strategy work? The re-ranking head is a lightweight network trained with binary cross-entropy loss during inference to reorder the top candidates from the initial retrieval stage, reducing false positives caused by high visual similarity between distant vineyard locations.
What dataset was used for evaluation? All experiments were performed using the TEMPO-VINE vineyard dataset, with additional ablation studies on the BLT dataset to validate the re-ranking contribution.
What were the key results in cross-season scenarios? The method improved Recall@1 from 16.4% to 34.3% in cross-season scenarios, demonstrating significant robustness to phenological changes in agricultural environments.