Researchers at Northeastern University developed JOIN, a three-phase system that enables a mobile robot to autonomously join a wheelchair-mounted arm already holding an object. In tests across four bimanual tasks, JOIN achieved 95% success rate, matching the reliability of full expert teleoperation while dramatically reducing operator effort compared to task-agnostic geometric approaches.
What the Researchers Built
JOIN is a full-stack system that solves bimanual joining: a condition where one robot arm has already committed to a grasp (the "anchor"), and a second mobile robot must approach, position itself, and grasp the same or a related object to perform a cooperative two-handed task. The system operates in three phases. First, the complement robot drives to a predefined observation pose and uses a vision-language model (Gemini Robotics-ER 1.6) to understand the task from a user command and visual scene. Second, it plans a base location and gripper orientation that maximize opposition force closure and directional manipulability—ensuring the complement’s grasp opposes the anchor’s forces and the robot can move along the task-critical direction. Third, the robot navigates to that pose and executes the grasp.
The hardware platform pairs a Kinova Gen3 wheelchair-mounted arm (anchor) with a Hello Robot Stretch 3 mobile manipulator (complement). Operator involvement is limited to a brief grasp correction after the autonomous pre-grasp phase, after which the two robots execute the collaborative motion under teleoperation. JOIN was tested on four daily living tasks: box lifting, bottle opening, stirring, and pouring.
Key Results
JOIN succeeded on 19 of 20 trials (95%), compared to 20/20 for full expert teleoperation and 14/20 (70%) for the AnyGrasp task-agnostic baseline. The system also produced grasp proposals that required significantly less operator correction than the baseline and matched teleoperation quality on graded metrics.
| Method | Success Rate | Mean Regrasps (Bottle Opening) | Mean Balls Spilled (Pouring) |
|---|---|---|---|
| Full Teleoperation | 100% (20/20) | 2.0 | 0.2 |
| JOIN | 95% (19/20) | 2.0 | 0.4 |
| AnyGrasp Baseline | 70% (14/20) | 3.2 | 7.3 |
JOIN’s grasp proposals lowered operator effort: on the box lift task, total operator correction time (grasp adjustment + execution) was markedly shorter than the baseline, approaching teleoperation efficiency. For bottle opening, JOIN matched teleoperation’s 2.0 regrasps versus the baseline’s 3.2. In pouring, the baseline spilled an average of 7.3 balls (including one trial that dropped 21), while JOIN spilled only 0.4—nearly indistinguishable from teleoperation’s 0.2.
How It Works
JOIN decomposes bimanual joining into three sequential phases that combine high-level reasoning with geometric precision.
Phase 1 – Plan: The complement robot first moves to a predefined observation position. Given a user command (e.g., “open the bottle”) and an RGB-D image of the scene, Gemini Robotics-ER 1.6 identifies task-relevant objects and infers the intended motion direction (e.g., twist, lift, stir). The system then evaluates candidate base poses and gripper orientations using a scoring function with two terms: opposition measures the force-closure quality between the anchor’s existing grasp and the candidate grasp—ensuring the two arms can resist forces along the task-relevant direction. Directional manipulability quantifies the complement’s ability to generate velocity along the required motion direction from that base pose. The highest-scoring configuration is selected.
Phase 2 – Drive: The mobile base navigates to the selected pose using its onboard sensors and local planner. The system uses Pinocchio for kinematic computations and does not rely on the VLM for any 3D geometric reasoning.
Phase 3 – Grasp: At the target pose, the complement executes the planned grasp. Control is then handed to a teleoperator who can make a small correction (average under 10 seconds) before the two arms execute the collaborative motion under teleoperation.
The key innovation is that the grasp and base pose are chosen for the task, not just for stable pick-up. This shift from geometry-only to task-aware planning accounts for the large performance gap over the AnyGrasp baseline.
Why This Matters for Robotics
For wheelchair users with limited upper-body mobility, current single-arm robot systems (like the Kinova JACO) can grasp and retrieve objects but cannot perform tasks that require two hands—opening a bottle while holding it, stirring a pot, or carrying a large box. JOIN enables a user to summon a second mobile robot that autonomously positions itself and provides the missing coordination.
The practical benefit is clear: the operator’s involvement shrinks to a brief grasp correction and a single continuous teleoperation motion, while the entire pre-grasp reasoning, planning, and navigation is handled autonomously. This brings assistive bimanual manipulation closer to a viable daily aid, especially in home environments where a mobile robot can be stored separately and called when needed.
For robotics engineers, JOIN’s architecture demonstrates a clean separation between semantic reasoning (VLM) and geometric calculation (analytical tools)—a pattern that avoids the common pitfall of relying on large models for precise spatial tasks. The problem formulation itself—"anchor-grasp-conditioned joining"—provides a template for other bimanual scenarios in manufacturing, healthcare, and household robotics. For current hardware options, see used cobots for sale that could serve as complement robots.
Limitations and Open Questions
The evaluation is preliminary: only five trials per condition, a single environment (one apartment), and one expert teleoperator who is also a paper author. No component ablations were run, so it is unclear whether each scoring term (opposition, directional manipulability) is individually necessary. The system only handles single-step joining—tasks requiring multiple re-grasps or dynamic object handoffs are not addressed. The observation poses were manually selected rather than optimized, and the VLM reasoning was limited to known objects and commands. Future work should include larger user studies, dynamic environments, and learned coordination policies that reduce the remaining gap to teleoperation speed.
Frequently Asked Questions
What exactly is bimanual joining? It’s a condition where one robot arm is already holding an object, and a second mobile robot must autonomously approach and grasp either the same object or a related one to perform a cooperative two-handed task.
How is JOIN different from typical robotic grasping systems? Most grasping systems only consider geometric stability (where to hold an object). JOIN also considers the task motion—it selects a grip and base pose that enable the two arms to move together effectively, such as twisting a lid or stirring.
What were the most important results? JOIN succeeded in 19 out of 20 trials (95%), nearly matching perfect expert teleoperation (20/20) and outperforming a task-agnostic baseline (14/20). Operator correction time was drastically reduced, and task quality (e.g., fewer spilled balls) was comparable to human teleoperation.
Can JOIN work in a real home today? The system has been demonstrated in a single apartment environment with a limited set of tasks. While the approach is promising, it has not yet been tested with multiple users, cluttered rooms, or dynamic obstacles—so more engineering and user studies are needed before daily deployment.
Conclusion
JOIN introduces a principled approach to bimanual joining that combines vision-language reasoning with analytical geometric planning. By shifting operator effort from the full task to a short grasp correction, it brings assistive bimanual robotics significantly closer to practical, everyday use. The problem formulation and system design offer a strong foundation for future work in collaborative manipulation for accessibility and beyond.