Robotic grasping is one of the fundamental capabilities required for autonomous object manipulation. In humanoid robots, this task requires the integration of visual perception, object localization, inverse kinematics (IK), and hand control. Even when a robot can accurately reach a target point in simulation, the same movement may fail on the physical platform due to the sim-to-real gap caused by mechanical inaccuracies, actuator imperfections, sensor noise, or simplified assumptions in the robot model.
We addressed this problem in our previous work by introducing a low-cost calibration method based on haptic feedback from a touchscreen. The robot touched a set of target points on the screen, and the recorded contact positions were used to train correction models that mapped real-world target positions to suitable simulated coordinates. This approach improved the reaching accuracy of the Neuro-Inspired COmpanion (NICO) humanoid robot. However, reaching a point is only the first step toward successful manipulation, since grasping additionally requires object detection, localization, hand positioning, and reliable execution of the grasp.
Related Work
Several studies address sim-to-real transfer directly in robotic grasping. James et al. used randomized-to-canonical adaptation for data-efficient sim-to-real grasping, while Gäde et al. applied domain randomization and domain adaptation to visuomotor grasping with the NICO humanoid robot. Compared to these approaches, our work does not train an end-to-end grasping policy, but uses explicit localization, IK, and calibration-based correction.
Materials and Methods
This work builds directly on our previous calibration approach. We use the same robot platform, simulation-control software, and calibrated correction models, which were adapted and further developed for the requirements of the tabletop grasping task.
Robotic Setup and Software
The experiments were performed with the humanoid robot NICO, using its right arm and four-finger child-sized hand. A paper grid provided a pseudo-ground truth for placing objects and evaluating grasping performance across the robot's reachable area. For visual perception, we used the robot's built-in stereo camera setup, which provides low-resolution fisheye images of 640x480 pixels.
Calibration Models
To reduce the discrepancy between target positions in the real workspace and corresponding positions used by the simulator and IK module, we used three calibration models. In the grasping task, these models transform the estimated object position on the table into a corrected target position for the robot hand.
The baseline model M1 uses a piecewise linear mapping based on the manually aligned simulation and interpolation of measured height values. M2 is a partially nonlinear neural-network correction model, where the horizontal target coordinates are predicted by a multilayer perceptron (MLP), while the height component is still obtained by interpolation. M3 is a fully nonlinear neural-network model that predicts the corrected three-dimensional target position directly. A detailed description of the calibration procedure, model training, and reaching evaluation is given in our previous work.
In this paper, the models are not retrained, but are used as correction modules within the grasping pipeline. This allows us to evaluate whether calibration models optimized for 2D reaching can also improve object grasping, where small residual positioning errors may have a larger effect on task success.
Hand Alignment Using Visual Feedback
In addition to the calibration models, we implemented a visual feedback method for aligning the robot hand with the detected object before grasping. First, the robot estimates the object position using the vision module and moves the hand to an approximate position above the object. Then, the vision module is used again to detect the robot hand and reconstruct its position using stereo vision.
The step size is gradually reduced as the distance decreases. This prevents large oscillations when the hand is already close to the object. The alignment is repeated until the horizontal distance between the detected hand position and the object position is below a predefined threshold (set to 1 cm). The hand is then lowered toward the object and the grasp is executed by closing the fingers.
To make hand detection more reliable, we also adjusted the hand pose during the alignment stage. The palm was rotated toward the camera so that the detector could observe the central part of the palm more consistently before estimating the hand position. This effectively prevented YOLO detection failures despite the lower general mAP for hands.
This method served as M4 in our comparison. In contrast to the M2 and M3 correction models, it did not use calibration-based target correction, but relied only on visual feedback to align the hand with the object before grasping.
Grasping Pipeline and Experimental Evaluation
For successful grasp execution, we also applied two task-specific corrections: the vertical target correction and the palm yaw correction. Suitable z values and yaw angles were manually measured at an irregular set of reference points distributed across the robot workspace. For arbitrary target positions, both values were then estimated using radial basis function (RBF) interpolation. The roll and pitch angles of the palm were kept fixed to maintain a top-down grasping pose.
The complete grasping pipeline starts with an image captured by the robot cameras. YOLO detector provides the object position in image coordinates, which is then used by stereo vision module to estimate depth. The detected point is transformed into robot's coordinate frame using the kinematic chain of the robot head. This real-world target position is then transformed into a simulated target position using one of the correction models. Finally, the yaw correction provides palm orientation, IK solver computes the required joint angles, and the movement is executed on the real robot through NicoMotion library.
We evaluated four grasping approaches using the models M1 to M4. The plush tomato was placed on 22 grid positions distributed across the reachable workspace of the robot arm. For each position and each model, three grasping attempts were performed, resulting in 66 attempts per model. In addition to the overall grasping success rate, we also evaluated the success rate inside the original neural-network correction area, which contained 10 target positions. This allowed us to compare both the performance in the calibrated region and the generalization of the models across the full tabletop workspace.
Calibration-Based Grasping Results
To verify this hypothesis, we visualized the predictions produced by the neural network in M3, including points outside the neural network (NN) correction area. The mean displacement between input target positions and predicted simulated positions was only 2.14 cm inside the correction area, but increased to 5.45 cm outside it. The predictions therefore confirm that M3 extrapolated poorly in a large part of the tabletop workspace, usually overshooting the required correction. The only exception was the region to the left of the original touchscreen area, where the predictions followed the same trend as inside the calibrated region, which also explains why several grasping attempts in this part of the workspace were successful.
Visual Feedback and Overall Comparison
The visual feedback hand alignment model was evaluated as an alternative to the calibration-based correction models. Unlike M2 and M3, it did not use the calibrated target correction, but relied only on visual feedback to align the detected hand position with the detected object position before grasping. This approach improved grasping performance across the full tabletop workspace. Successful grasps were no longer concentrated only inside the neural-network correction area, but also appeared in regions outside the original touchscreen area. Overall, this model achieved a 72.7% success rate, corresponding to 48 successful grasps out of 66 attempts.
The M2 model achieved lower performance than M3, which is consistent with the reaching results reported in our previous work. Inside the neural-network correction area, the best result was obtained by M3 with 96.7% success rate. Across the whole tested workspace, the best result was obtained by the visual feedback model with 72.7% success rate.
Discussion
Direct comparison with other robotic grasping studies is difficult, since platforms, objects, sensors, and evaluation protocols differ substantially. The most comparable work is where sim-to-real grasping on NICO robot was also evaluated, but with side grasping of a cylindrical object using the left hand. Their best model achieved an 80.3% success rate using domain adaptation and inherent IK. Our visual feedback model achieved 72.7% overall, while the fully nonlinear calibration model M3 reached 96.7% inside the calibrated NN correction area.
The main limitation of the visual feedback model was stereo vision-based hand localization. In several positions, the hand position was strongly overestimated when the background behind the hand was too far from the robot. This reduced the overall success rate, although performance remained high when hand localization was correct. Improving hand segmentation or depth estimation would likely further improve the model.
In summary, we implemented a low-cost grasping pipeline combining object detection, stereo vision localization, and calibrated IK control without RGB-D cameras, motion capture, or external tracking. Although task-specific corrections were used to compensate for mechanical inaccuracies, the underlying methodology remains generalizable. The results show that haptic calibration developed for 2D reaching can also improve object grasping, while visual feedback provides an effective alternative when calibration-based correction is not used.
We thank anonymous reviewers for detailed feedback. I.F. was supported by Slovak Research and Development Agency, project APVV-21-0105.
Frequently Asked Questions
What is the sim-to-real gap and why does it affect NICO's grasping? The sim-to-real gap refers to differences between simulation and physical hardware caused by mechanical inaccuracies, actuator imperfections, and sensor noise. These discrepancies cause movements that work perfectly in simulation to fail on the real robot.
How do the calibration models M1, M2, and M3 differ from each other? M1 uses a simple piecewise linear interpolation, M2 combines a neural network for horizontal correction with interpolation for height, and M3 is a fully nonlinear neural network that predicts all three target coordinates directly from the real-world position.
Why did the visual feedback model M4 achieve better overall success than calibration models? M4 used iterative visual alignment of the hand with the object before grasping, which worked across the entire workspace. Calibration models M2 and M3 only performed well inside the limited area where they were trained with touchscreen data.
What is the main limitation of the visual feedback approach? Stereo vision-based hand localization sometimes overestimates hand position when the background behind the hand is too far from the robot. This causes alignment errors and reduces grasping success rate in those positions.