Researchers have created a complete system that lets an operator remotely control both the arms and legs of a miniature humanoid robot using a VR headset, hand controllers, and a treadmill. By combining real-time motion retargeting with a reinforcement learning (RL) walking policy, the system achieves full-body teleoperation without requiring manual balancing or gait switching.
What the Researchers Built
The team built a complete software and hardware framework for tele-loco-manipulation – controlling a humanoid robot’s locomotion and manipulation simultaneously through a virtual reality interface. The hardware setup uses a VIVE Pro 2 headset to track the operator’s head movements and two VIVE controllers for hand poses, all optically tracked by four SteamVR Basestation 2.0s. A KAT Walk C2+ VR treadmill captures the operator’s walking motions (though this feature is reserved for future study). Everything runs on a high‑end PC with an i9 CPU and RTX 4090 GPU to minimise VR latency.
The target robot is the ROBOTIS OP3, a miniature humanoid with 20 revolute degrees of freedom (6 DoF per leg, 3 DoF per arm, and a 2 DoF pan‑tilt head). Its DYNAMIXEL XM430‑W350‑R servos are controlled via a data‑driven actuator model developed by the researchers. The robot’s original camera was replaced with a VR180 dual‑fisheye lens USB camera to give the operator an immersive first‑person view. On the software side, the system combines a full‑body PD‑based impedance controller for compliant manipulation with an RL‑based locomotion policy that handles walking and standing balance. The entire pipeline was validated on the real OP3 hardware.
Key Results
The researchers conducted two types of experiments to validate the system:
- Telemanipulation validation: The operator traced a circle rapidly with both arms ten times while the VR controller positions and the robot’s actual end‑effector positions were recorded at 50 Hz. This measured motion‑tracking latency and accuracy of the arm retargeting pipeline.
- Locomotion validation: Random whole‑body velocity commands were sent to the robot every three seconds over an 80‑second trial. The normalised projected gravity – a metric for bipedal stability – was recorded alongside the commanded velocities to assess how well the RL policy maintains balance under changing speeds.
The paper reports that the system successfully demonstrated bimanual tele‑loco‑manipulation in a sample trial (available in the supplemental video). While exact numerical benchmarks are not provided in the abstract, the experiments confirm that the integrated framework can track operator hand motions in real time and sustain stable locomotion under varying commands. The data‑driven actuator model proved essential for achieving reliable torque control on the DYNAMIXEL servos.
How It Works
The system works in three interconnected layers:
- VR motion capture: The VIVE Pro 2 HMD and controllers report 6‑DoF poses at up to 90 Hz. The operator’s arm and head positions are retargeted to the robot’s corresponding joint angles using inverse kinematics. The VR180 camera feed is streamed back to the HMD for telepresence.
- Actuator modeling and impedance control: Because the DYNAMIXEL XM430 servos have limited built‑in torque control, the team derived a data‑driven model of each actuator (mapping desired torque to commanded current). A full‑body PD‑based impedance controller then translates desired end‑effector forces into joint torques, allowing the robot to comply physically with external disturbances while following the operator’s commands.
- RL locomotion policy: A reinforcement learning policy was trained in simulation to output joint torques that make the robot follow a desired linear and angular velocity while staying upright. The policy was transferred to the real robot using system identification on the actuators (the data‑driven model). The RL policy runs at the same control loop as the impedance controller, so manipulation and locomotion commands are blended at the torque level.
The software architecture is modular: the VR interface sends high‑level commands, the RL policy handles walking, and the impedance controller manages arm compliance. All three run concurrently on the OP3’s onboard i7 NUC and OpenCR board.
Why This Matters for Robotics
This work demonstrates that full‑body teleoperation of a miniature humanoid is feasible using off‑the‑shelf VR hardware and learning‑based control. For robotics professionals, this opens up several practical use cases:
- Remote inspection and manipulation in hazardous environments (e.g., nuclear decommissioning, disaster response) where a humanoid form factor is needed to navigate stairs or tight spaces.
- Telepresence with physical interaction – the impedance controller ensures the robot can safely contact objects while the operator remains immersed.
- Scalable training – the RL locomotion policy can be improved over time without changing the hardware, potentially allowing the same framework to control different humanoid platforms.
The system is particularly relevant for small humanoids like the ROBOTIS OP3, which are affordable and widely used in research labs and education. You can explore similar humanoids on Robot Overflow’s humanoid robot listings or find used collaborative robot arms for related manipulation tasks at used cobots for sale.
Limitations and Open Questions
The system has several acknowledged limitations:
- The robot is tethered to wall power, limiting autonomy and field deployment.
- The locomotion policy has only been validated at moderate walking speeds; rapid turns or slopes may destabilise it.
- VR motion tracking can suffer from occlusion or jitter, which propagates to the robot’s end‑effector.
- The actuator model was derived for the specific DYNAMIXEL XM430 servos; porting to other actuators would require re‑modelling.
Open questions include how well the system scales to larger humanoids, whether the VR treadmill can replace operator body tracking for more natural locomotion, and how to handle multi‑contact tasks (e.g., climbing stairs) without manual mode switching.
Frequently Asked Questions
What hardware is required to use this system? A VIVE Pro 2 headset, two VIVE controllers, four SteamVR Basestation 2.0s, a KAT Walk C2+ treadmill (optional), and a ROBOTIS OP3 humanoid with its onboard computer.
Can the operator control the robot’s walking speed and direction? Yes – the RL locomotion policy accepts continuous linear and angular velocity commands, which can be generated by the operator’s body motion or via a joystick input (in the current setup, they are sent programmatically).
How is the robot’s balance maintained during manipulation? The RL policy handles balance by generating joint torques that counteract the effects of arm movements and external forces. The impedance controller adds compliance, so the robot does not become rigid during contact.
Is the system ready for commercial deployment? Not yet – it is a research prototype that requires a dedicated PC, a tethered power supply, and careful calibration. However, the software architecture is modular and could be adapted to other humanoid platforms.
Conclusion
This research proves that combining VR telemanipulation with a learned locomotion policy can deliver real‑time full‑body control of a miniature humanoid robot. The open‑source‑style actuator modelling and impedance control provide a solid foundation for future work on portable, lower‑cost telepresence robots. Further integration of the VR treadmill and dynamic balancing under load will be key next steps.