Imitation of arm gestures from a single RGB camera is achieved by combining MediaPipe landmark extraction with analytical geometry, adapted to the semi-humanoid robot NICO. The proposed method requires no training data, is computationally efficient, and provides an interpretable mapping from visual observations to robot motor commands.
Introduction
Overall, existing work demonstrates that imitation can be achieved using both analytical and learning-based approaches. Learning-based methods offer flexibility and generalization but at the cost of complexity, while analytical approaches provide efficiency and transparency but often require adaptation to specific kinematic structures. Moreover, prior studies either focus on simplified gestures or depend on specialized sensing hardware.
In this context, the present work extends analytical geometry-based methods to a broader set of arm gestures using only RGB input. By leveraging MediaPipe for landmark extraction and adapting geometric formulations to the kinematics of the NICO robot, the proposed approach achieves meaningful imitation without reliance on expensive sensors or computationally intensive learning models.
Related Work
With the advancement of computer vision, systems have emerged for extracting human skeletons from RGB images. One of the most well-known such systems is OpenPose, which is designed for multi-user 2D pose estimation, using two CNNs to predict body parts and their associations.
More recent solutions, such as MediaPipe, enable the real-time extraction of pseudo-3D landmarks using standard RGB cameras. Thanks to its low latency, MediaPipe has become a popular solution for human-robot interaction and gesture-based systems.
With the emergence of pose-estimation systems, vision-based human motion imitation became more feasible. Instead of relying on wearable sensors or motion-capture systems, robots can now infer human body configuration directly from RGB images and reproduce the observed motion. This development motivated both analytical geometry-based methods and learning-based imitation approaches.
Early neural-network-based approaches to robot imitation were explored by Durdu et al., who used artificial neural networks to classify human arm movements collected from joint-mounted sensors and reproduce them on a robotic arm platform.
Self-supervised approaches were explored by Lúčny et al., where robots learn imitation through associations between visual observations and their own motor states. These methods reduce the need for manually annotated datasets and demonstrate promising generalization capabilities. Although learning-based methods demonstrate high flexibility and the ability to generalize, they typically require large-scale datasets, training procedures, and substantial computational resources.
A number of studies use analytical geometry to transform landmark coordinates into joint angles without relying on learning-based methods. Altayeb used MediaPipe to extract hand landmarks, while joint angles were computed using direct geometric relations for further robotic hand control. Their study demonstrates that analytical methods can achieve high accuracy without relying on learning-based approaches. The reported accuracy of approximately 96 percent indicates that analytical approaches are promising, but their applicability to more complex movements remains an open question.
Another relevant study presents a system based on depth-camera input. In this work, joint angles are derived using geometric relations and mapped to the NAO robot platform. The authors report an imitation accuracy of approximately 95 percent for arm motion imitation, demonstrating that analytical methods can be effective when accurate 3D data is available. However, the reliance on depth-sensing hardware increases system cost and complexity.
In contrast, the present work applies a similar analytical framework to the humanoid robot NICO using only RGB camera input. Since the kinematic structure of NICO differs from that of the NAO robot, the corresponding angle formulations must also be adapted. In particular, the proposed system includes estimation of forearm rotation and wrist flexion, while NICO does not provide the elbow roll joint available on the NAO platform. Furthermore, this work incorporates MediaPipe hand landmarks, which provide additional information for estimating wrist flexion and forearm rotation. Head motion imitation is outside the scope of the current implementation and is therefore not addressed experimentally.
Overall, existing analytical approaches demonstrate that promising pose imitation is achievable without learning-based models. However, prior work either focuses on simplified gesture sets or depends on specialized hardware. Thus, the proposed approach does not require a training stage, uses only monocular RGB input, supports full arm and hand reconstruction, and is tailored to the kinematics of the NICO robot.
Materials and Methods
Here we describe the hardware and software used, as well as the processing pipeline for gesture imitation.
3D Coordinates Extraction
A key limitation of the proposed system is the use of a single RGB camera without additional sensing modalities such as depth sensors. To estimate 3D human pose under this constraint, the MediaPipe framework is employed. MediaPipe provides models for extracting both pose and hand landmarks directly from RGB images.
In this work, we use landmarks 11–24, focusing on arm motion. The approach can be extended to full-body imitation, which remains an interesting direction for future investigation.
The accuracy of MediaPipe outputs has been evaluated in prior work. In that study, four subjects performed multiple physical exercises recorded from different viewpoints, with ground truth provided by a motion capture system (MTw Awinda). The results indicate that estimation accuracy strongly depends on the viewing angle, with increased error under occlusions. Additionally, joint angle estimation was found to be more reliable than absolute body measurements. These findings are consistent with observations in this work, where occlusions and non-optimal viewpoints lead to decreased precision.
Angle Mapping to NICO
The joint angles reconstructed from the geometric model cannot be used directly as motor commands because each NICO joint has its own admissible range and motor coordinate system. Therefore, each analytically reconstructed angle was linearly mapped to the corresponding motor range.
Since the motor positions at the interval boundaries are specified explicitly, the same equation naturally handles both matching and reversed motor orientations. Finally, the computed motor position is rounded to the nearest integer before being sent to the controller.
Conclusion
This paper presented an analytical pipeline for monocular RGB-based imitation of human arm gestures by the semi-humanoid robot NICO. The proposed method combines MediaPipe pose and hand landmark extraction with geometric joint-angle reconstruction and an angle-mapping procedure tailored to NICO's kinematic structure. Unlike learning-based imitation methods, the proposed approach requires no training data, is computationally efficient, and provides an interpretable mapping from visual observations to robot motor commands.
Experimental evaluation on a dataset of predefined arm poses demonstrated that the method reconstructs shoulder pitch, elbow bend, and wrist flexion with moderate accuracy, while shoulder roll and especially forearm rotation remain more challenging. The results further showed that incorporating MediaPipe hand landmarks substantially improves forearm rotation estimation compared with using pose landmarks alone, although this joint continues to represent the primary source of reconstruction error. Reconstruction accuracy was relatively consistent across participants and viewing angles under the controlled experimental conditions, whereas the performed gesture had a much greater influence on performance.
The presented method provides a proof of concept for analytical monocular RGB-based gesture imitation using MediaPipe landmarks and geometric reconstruction. While the approach achieves encouraging results for several arm joints under controlled conditions, further improvements and validation on larger and more diverse datasets are required before it can be considered a robust solution for general gesture imitation. Future work will focus on improving forearm rotation estimation, incorporating temporal smoothing, extending the system to dynamic gestures, and evaluating the method on more diverse subjects and viewpoints.
The authors are thankful to Branislav Zigo for technical assistance. This research was supported by Horizon Europe MSCA project TRAIL, no. 101072488 and by Slovak Research and Development Agency, project APVV-21-0105.
Frequently Asked Questions
How does the NICO robot imitate arm gestures without training data? The robot uses MediaPipe to extract human pose and hand landmarks from a single RGB image, then applies analytical geometry formulas to compute joint angles that are mapped directly to NICO's motors, requiring no machine learning training.
What joints can the system reconstruct and which are most challenging? The method reconstructs shoulder pitch and roll, elbow bend, forearm rotation, and wrist flexion. Forearm rotation is the most challenging joint, with the largest reconstruction errors.
How do MediaPipe hand landmarks improve imitation accuracy? Incorporating hand landmarks substantially improves forearm rotation estimation compared to using only pose landmarks, because they provide additional geometric constraints for computing the rotation angle.
Does the system work from any camera viewing angle? Reconstruction accuracy depends on the viewing angle. MediaPipe's landmark estimation error increases under occlusions and non-optimal viewpoints, which directly affects joint angle accuracy.