A robot navigation system called H2INT lets a mobile robot move through dense crowds using only relative pedestrian positions, without gaze, body orientation, velocity estimates, or explicit labels saying who is reacting. Its two-stage gated Transformer and recurrent memory handle human-human and human-robot interactions, while a responsiveness curriculum improves training stability and a LiDAR-equipped robot reaches goals in an indoor venue.
What the Researchers Built
H2INT is a navigation policy for robots operating around people whose behavior is difficult to predict. Rather than assuming that every pedestrian notices the robot and reacts in the same way, the system models a mixture of behaviors: some people adjust their path because of the robot, while others continue walking independently. Each pedestrian’s response tendency remains hidden from the robot and persists over time, making the problem closer to real crowd behavior.
The robot receives relative positions from its sensors. It does not receive a person’s gaze direction, body orientation, attention label, or estimated velocity. This sparse input forces the policy to infer interaction patterns from how the crowd changes over time.
The central model contains two gated Transformer stages. The first stage builds an initial representation of the robot’s relationships with nearby pedestrians. The second stage refines that representation by combining the interaction features already extracted. GRU-style gates control how much new information enters each layer, instead of relying on fixed residual connections.
A recurrent policy core then carries information between time steps. Together, the Transformer and recurrent memory support direct action selection in a variable-size crowd. The system outputs movement commands through an actor-critic reinforcement-learning policy trained with PPO.
Key Results
H2INT was evaluated in the CrowdNav Circle Crossing setting and tested against three structurally different crowd-flow modes that were not used during training. The evaluation focused on whether the robot could preserve navigation performance when pedestrian responsiveness and crowd density changed.
The complete H2INT model produced the most consistent success-rate trend across the tested response regimes and crowd sizes. A single-stage Transformer performed worse in every tested setting, supporting the value of refining interaction features in two stages. Replacing attention with mean pooling caused the largest degradation when pedestrians were least responsive, showing that simple feature averaging loses important relationships within the crowd.
Standard Transformer residual connections also became less reliable at the highest crowd density. The gated updates helped the network control information flow as interactions became more complicated.
The training curriculum produced faster average-return improvement with lower variance than abruptly training at a fixed difficulty. This curriculum gradually increased the share of pedestrians who ignored the robot, helping the policy move from cooperative scenarios toward more uncertain ones.
A real-world test transferred the policy to a mobile robot using LiDAR pedestrian detections. The robot reached its goal without collision while people crossed its path, demonstrating deployment feasibility rather than a statistically broad field benchmark.
How It Works
H2INT converts each observed pedestrian relationship into a token. A token is a compact feature vector that represents one element of the scene. The robot itself gets a dedicated token, while each nearby person is represented through a robot-relative edge token containing geometric information such as relative position. Type embeddings distinguish the robot token from pedestrian-relationship tokens, and positional encoding preserves structural identity.
Self-attention lets each pedestrian token exchange information with the others. That means the representation of one person can reflect the surrounding crowd rather than treating every pedestrian as an isolated obstacle. This provides an indirect form of human-human interaction modeling without requiring explicit pedestrian velocity estimates.
The first gated Transformer stage performs an initial relational update. After attention and the feed-forward network, GRU-style residual gates decide how strongly the new features should replace or modify the previous representation. The second stage repeats this process on the already enriched features, producing a more refined robot-centered interaction summary.
A masking mechanism prevents padded entries from representing nonexistent pedestrians when the crowd contains fewer people than the model’s configured maximum. The computation still scales with that padded sequence length, so very large sensing ranges or crowd limits can increase runtime.
The refined context vector enters a recurrent actor-critic network. Its GRU stores temporal information, helping the policy infer motion trends from successive position observations. The actor outputs a probability distribution over movement actions, while the critic estimates the expected reward. PPO and generalized advantage estimation train both the Transformer and recurrent policy together.
Unlike trajectory-prediction planners, H2INT does not first roll out a future path for every pedestrian. It maps the current robot-centered observation directly to an action distribution, keeping online decision-making comparatively compact.
Why This Matters for Robotics
Crowd navigation fails when a robot treats every person as equally predictable. In a station, exhibition hall, hospital, or warehouse, some people make room, some do not notice the robot, and others change direction because of unrelated pedestrians. H2INT addresses this variability without requiring cameras that estimate gaze or body orientation.
That sparse sensing interface is valuable for practical deployments. A LiDAR system can provide relative positions while avoiding the privacy, lighting, and calibration challenges associated with interpreting faces or body posture. The real-robot trial also shows that the policy can operate with noisy detections, partial occlusion, and motion patterns absent from simulation.
The approach is particularly relevant to warehouse robots that share aisles or staging areas with workers. It could also support service robots and delivery platforms moving through public spaces, provided their control systems can enforce safety limits around the learned policy.
The most important design lesson is that perception and behavior should be tested together. Better network capacity alone cannot solve a simulator that assumes every pedestrian responds identically.
Limitations and Open Questions
The reported real-world evaluation is a demonstration, not a broad statistical study. A single indoor venue and a successful collision-free run cannot establish reliability across different buildings, crowd densities, sensor layouts, or pedestrian cultures.
The behavior model is intentionally compact. It represents whether pedestrians respond to the robot, but not richer causes of that response, such as distance, relative speed, body orientation, group membership, or a person’s intended destination. Those omitted signals could improve prediction while also increasing sensing and modeling complexity.
The architecture also uses dense attention over a padded crowd representation. Runtime and memory could become limiting factors in very large crowds or on small onboard computers. Future testing needs repeatable measures for clearance, comfort, intervention rate, computation time, and performance under sensor dropouts.
Frequently Asked Questions
What does H2INT observe? It observes robot-centered relative pedestrian positions over time, rather than gaze, body orientation, velocity estimates, or explicit attention labels.
Does H2INT identify whether a person is paying attention? No. The system keeps pedestrian responsiveness latent and learns its behavioral consequences from motion feedback.
Why does H2INT use two Transformer stages? The first stage builds interaction features and the second refines them, which performs better than the tested single-stage alternative.
Was H2INT tested on a physical robot? Yes. A LiDAR-equipped mobile robot used the policy in an indoor exhibition venue and reached its goal without collision during the reported trial.
Conclusion
H2INT combines gated relational attention with recurrent memory to help robots navigate crowds where pedestrian responses are heterogeneous and hidden. Its simulation comparisons, unseen-flow testing, and physical-robot demonstration show a practical direction for sparse-sensor social navigation, while broader field evaluation remains necessary.