A major reason for navigation errors is the inability to associate motion predictions with their potential outcomes. Most methods directly predict continuous waypoints from visual observations and language instructions. Since their training objective penalizes only waypoint error, the model can hardly anticipate the execution outcomes of its own actions, increasing the risk of generating trajectories that may be unsafe in a 3D environment.
We experimentally verify the effectiveness of ProtoVis-Nav on OpenUAV and HUGE-Bench. Results show that ProtoVis-Nav substantially improves navigation performance over strong baselines and leads to more reliable continuous UAV navigation.
Overall Pipeline
ProtoVis-Nav uses Qwen3VL-2B-Instruct as its vision-language model (VLM) backbone. The input consists of dual-view RGB observations (front-view and down-view images), a language instruction, and the current UAV state, including the previous displacement and current position. The Qwen3VL backbone is kept frozen, while LoRA adapters inserted into the backbone and newly added output heads are trained.
For visual imagination, the tokens predict feature prototypes, which are combined through softmax-normalized linear weighting to produce an outcome-associated local visual representation of the target local region. This encourages the model to anticipate what the relevant visual context should look like before predicting the next action.
Finally, the token considers the tokens as context within the VLM. The waypoint head combines its hidden state with UAV kinematic information to predict the next waypoint.
Variant Study
Variant studies on the Test Seen split analyze three design choices of ProtoVis-Nav: the visual imagination target, the PVI head design, and the number of tokens.
PVI Spatial Visualization and Waypoint Token Attention
Spatial visualization
Two examples are visualized by overlaying the PVI attention map on the current image. The cruise and landing examples connect the intended next UAV movement to local visual regions in the front and down views, respectively. In both cases, the attention peak lies within or near the projected target region. This suggests that PVI focuses on visual regions relevant to the next intended UAV movement.
Waypoint token attention
The waypoint token’s causal attention is analyzed over 1,000 frames. Attention matrices are averaged across frames to summarize interactions between image, PVI, and waypoint tokens. The waypoint token’s attention concentrates on the PVI tokens. Through causal attention, it can combine this movement-related visual context with the instruction and UAV state. Together, the visualizations and analysis suggest that PVI identifies locally action-relevant visual evidence and influences the waypoint token’s representation through attention.
Qualitative Analysis
Qualitative examples illustrate how ProtoVis-Nav navigates toward the target. In one example, the instruction describes a white truck parked on an empty street surrounded by multi-story stone buildings. The UAV gradually moves along the street, adjusts its viewpoint across consecutive observations, and finally stops near the target truck. The sequential observations show that ProtoVis-Nav maintains target-relevant visual context for waypoint prediction under changing viewpoints. This result is consistent with the motivation that prototypical visual features provide useful context for waypoint prediction.
Limitations and Social Impact
Despite the improvements, ProtoVis-Nav still faces challenges in complex aerial environments. Based on the authors’ observations, most failure cases are caused by collisions, especially with trees and thin obstacles such as branches, poles, and wires. These obstacles often occupy only a small image region and may be partially occluded or visually ambiguous. This makes obstacle boundaries and available clearance difficult to estimate from RGB observations.
Future work will explore stronger obstacle-aware perception and planning mechanisms, including depth or LiDAR inputs, explicit collision-risk modeling, and anti-collision trajectory refinement for safer UAV navigation. The closed-loop evaluation is conducted in simulation. Future work will extend validation to real-world UAV flights and hardware-in-the-loop experiments to assess closed-loop navigation performance under physical sensing and control conditions.
ProtoVis-Nav may benefit applications such as search-and-rescue, disaster response, and infrastructure inspection by improving UAV navigation from language instructions. At the same time, UAV navigation systems may raise safety, privacy, and security concerns if deployed without sufficient safeguards, and should therefore be used with human oversight and deployment constraints.
Conclusion
ProtoVis-Nav is a Prototypical Visual Imagination framework for UAV vision-language navigation. Its key idea is to ground waypoint prediction in local visual context associated with the next intended movement, which is especially important for UAVs, where small 3D motion errors may lead to unsafe navigation behavior.
ProtoVis-Nav imagines an outcome-associated local visual representation for the intended next UAV movement. Specifically, it predicts feature prototypes and linearly combines them to estimate the target-region visual feature, while the prototype token representations provide context for waypoint prediction. This design promotes better alignment between visual context and waypoint outputs. Experiments on OpenUAV and HUGE-Bench demonstrate the effectiveness of ProtoVis-Nav.
Acknowledgments and Disclosure of Funding
This paper was sponsored by the National Natural Science Foundation of China (No. U25A20434).
Stop Supervision
A simple stop supervision strategy is designed to exploit the VLM’s internal grounding ability to decide when the UAV should terminate navigation. The stop head is supervised with a distance-aware soft label, encouraging the VLM to ground the stopping decision in the current visual observation and its distance to the target.
Given the output hidden state of the token, a linear layer followed by a sigmoid activation predicts the stopping probability.
The stop probability is predicted from the hidden state of the token.
During training, the supervision label is constructed according to the distance between the current UAV position and the target position.
The soft supervision label depends on the distance between the UAV and the target.
This soft label assigns a value of 0.5 when the UAV is 20 meters away from the target. As the UAV gets closer to the target, the label increases toward 1; when it is farther away, the label decreases toward 0. The stop head is trained with binary cross-entropy.
During inference, the UAV sends a stop signal according to the predicted stopping probability.
The stopping decision is made using the predicted stop probability.
This design provides a soft and distance-aware stopping objective, encouraging the model to stop only when its visual-language grounding indicates that the UAV is sufficiently close to the target.
Frequently Asked Questions
What is ProtoVis-Nav? It is a prototypical visual imagination framework for UAV vision-language navigation.
What inputs does ProtoVis-Nav use? It uses front-view and down-view RGB images, a language instruction, and the current UAV state, including previous displacement and current position.
How does visual imagination support waypoint prediction? The tokens predict feature prototypes that are combined into an outcome-associated local visual representation. This representation provides context for predicting the next waypoint.
What limitations does the paper identify? Most observed failures involve collisions, particularly with trees and thin obstacles such as branches, poles, and wires. The evaluation is in simulation, and real-world UAV validation remains future work.