HEIR pairs a person’s first-person video, gaze, and spoken requests with a mobile robot’s views and actions, creating a benchmark for household human-robot interaction. It evaluates intent understanding, planning, navigation, and manipulation, and its real-robot tests report pick-and-place success from 36.67% to 68.33%—a concrete measure of how difficult it remains for robots to act on what people mean.
What Did the Researchers Build?
HEIR is a dataset and benchmark for studying robots that respond to people’s requests in household settings. Instead of recording only the robot’s camera feeds and movements, it also captures the person’s perspective: egocentric video (video recorded from the wearer’s point of view), speech, and eye gaze. The robot-side recordings include views from head and wrist cameras, movement and gripper commands, and measurements of the robot’s physical state.
The data comes from staged daily-living, kitchen, and home-office scenarios, including fetching objects, handing them to someone, moving them, and working together on a task. A mobile, dual-arm robot was remotely operated as participants made requests in their own words. The resulting sessions contain a sequence of requests, each linked to the robot’s response and its navigation and manipulation subtasks. Because requests in the same session share an evolving environment, later instructions can refer to things that happened earlier.
Annotations connect each request to its intended task, objects, destination or recipient, and expected task structure. They also record what the robot actually did and when. Reviewers check the labels against the recordings, while language-model assistance helps describe subtasks and their completion conditions.
What Results Did HEIR Report?
The clearest reported real-robot comparison concerns pick-and-place manipulation. Three tested systems completed 20 tasks, with three trials per task. Lingbot-VLA2 achieved the highest success rate at 68.33%, followed by FastWAM at 60.00%. A third evaluated model recorded 36.67%; its name is not available in the reported text.
| Model | Real-robot pick-and-place success |
|---|---|
| Lingbot-VLA2 | 68.33% |
| FastWAM | 60.00% |
| Third evaluated model | 36.67% |
The ranking matched the systems’ relative performance in offline action prediction, where models are assessed against recorded examples rather than deployed on the robot. That agreement suggests offline evaluation can help narrow down manipulation models before running more costly physical trials.
HEIR also evaluates human-intent understanding, planning, and navigation, but the reported material does not include numerical scores for those comparisons. Its overall analysis highlights the value of human-centered cues while also identifying a practical challenge: errors can build up when intent understanding, planning, and robot actions are combined in a long task. The findings therefore offer both a useful performance reference and a reminder that success on one component does not guarantee success across an entire interaction.
How Does the HEIR Benchmark Work?
HEIR breaks a household interaction into a hierarchy. A session contains ordered request-level episodes; each episode links a spoken request to the robot’s execution. That execution is divided into navigation and manipulation subtasks, such as moving to a location and then picking up or handing over an object. The number and order of subtasks can change depending on the request and the robot’s current state.
The benchmark first tests whether a system can interpret the person’s intent. An intent reasoner processes each request before the robot moves. It uses the request and relevant human-centered cues to identify the task, objects, goal entities—such as a recipient or destination—and a nominal plan. That plan describes the expected task structure; it is context for the next stage, not a fixed sequence the robot must blindly follow. The planner selects the realized subtasks online, based on the robot’s state.
For navigation, a vision-language navigation model combines a natural-language subgoal with visual and spatial context to predict movement commands. A LiDAR sensor also supports collision avoidance. For manipulation, a vision-language-action model or world-action model predicts robot movements and gripper actions. These models are evaluated on recorded data, with measures including action accuracy and errors in predicted joint motion and end-effector position.
The real-robot evaluation adds a physical test: no human intervention is allowed, and a request counts as successful only if the robot completes the whole request. That distinction matters. A model can predict actions that resemble recorded demonstrations yet still struggle to complete a sequence of subtasks in the real world.
Why Does HEIR Matter for Robotics?
Many robots are designed to follow a specified task while treating people mainly as nearby obstacles or sources of instructions. HEIR focuses on a harder, more natural situation: people refer to objects, destinations, and earlier events in context, and their gestures or gaze can clarify what they mean. A robot that uses those cues could respond more flexibly than one that relies on a short, explicit command alone.
The benchmark also gives engineers a way to inspect different links in the interaction chain. If an entire household task fails, separate evaluations can help reveal whether the breakdown came from interpreting the request, choosing subtasks, navigating, or manipulating an object. Comparing offline predictions with physical trials can help identify promising policies before committing to repeated robot testing.
For operations teams assessing platforms for service or assistance work, HEIR highlights capabilities that go beyond basic mobility or picking: understanding references, handling changing task order, and completing a request without intervention. Teams exploring available hardware can browse used robots for sale, while keeping in mind that platform capabilities alone do not solve the interaction problem. The benchmark offers a basis for asking how well a robot works with people, not just how well it performs an isolated motion.
What Are HEIR’s Limitations and Open Questions?
HEIR’s human-centered signals include gaze measured with eye-tracking glasses and video from the person’s point of view. That makes the signals valuable for research, but raises a deployment question: household robots cannot assume every user will wear recording equipment. Future systems need to establish which cues can be inferred from the robot’s own cameras, speech, and interaction history.
The scenarios are staged, and a separate operator teleoperates the robot during data collection. This produces useful examples of household interaction, but demonstrated behavior may differ from what an autonomous system can reliably produce in an unfamiliar home. The reported real-robot success rates cover pick-and-place tasks; they do not establish end-to-end success across all the benchmark’s long-horizon request types. Further evaluation across varied homes, people, and complete tasks would clarify how well the approach transfers.
What Questions Do Readers Ask About HEIR?
What does HEIR stand for? The paper uses HEIR as the name of its dataset and benchmark, Harness Egocentric Intent for Human-Robot Interactions.
What information does HEIR collect about people? It records egocentric video, speech, and measured gaze, aligning those signals with the robot’s views, state, and actions.
Which robot tasks does HEIR evaluate? It covers intent understanding and subtask planning, as well as navigation and manipulation in household interactions.
Does HEIR show that a robot can complete a whole household request autonomously? The benchmark includes real-robot testing, but the reported success rates here are for pick-and-place tasks, not all long-horizon household requests.
What Is the Main Takeaway?
HEIR makes human intent and context part of the robotics benchmark, linking what a person says and attends to with what a robot plans and does. Its results show that physical manipulation is measurable today, while reliable long-horizon interaction still requires better coordination across the full task.