Online Jacobian Estimation and Visual Servoing
Uncalibrated visual servoing learns the differential map between sensed features and actuation online, with no camera or kinematic calibration. Hosoda and Asada introduced the canonical recursive-least-squares (RLS) estimator with a forgetting factor, the direct ancestor of this update. Qian and Su estimated the image Jacobian online with a Kalman filter.
Recent attempts have extended this to a more generalized descriptor of the Jacobian, obtaining a dense, per-pixel image Jacobian or a visuomotor Jacobian field defined on all 3D points along the robot.
However, the visuomotor Jacobian field required extensive data to compute, around 2–3 hours of robot motion from a multi-view RGBD camera setup. These past methods showed no or only limited application to tool manipulation: bimanual broom-tip manipulation has been demonstrated, and Jacobian-field-based control has been applied to grasp a tool and push an apple along a table in one demonstration.
Prior work has also estimated the inverse of the Jacobian instead of the Jacobian itself, with little or no a priori information, and applied it to a writing task in air. By directly estimating the inverse Jacobian, a direct map from task space to command, or joint, space can be obtained, eliminating the need to computationally invert the Jacobian matrix. However, because only its inverse is estimated, it is not clear how to compute and utilize the nullspace of the Jacobian, limiting extensibility to redundant structures.
In-Hand Writing as a Benchmark
We show broad adaptability of the method: after approximately 18 seconds of initialization, the physical ORCA hand writes the whole alphabet while continuously updating its Jacobian estimate. We also demonstrate the same estimator-controller formulation on two other hands in simulation, with the code released as an open-source implementation.
Perception for Pen-Tip Tracking in the Paper Frame
A webcam tracks an ArUco marker attached to the pen and a board of three markers taped to the desk, defining the paper frame. The pen-tip position is recovered using a fixed marker-to-tip offset and transformed into the paper frame, which is used for all downstream estimation and control. The webcam view includes the marker detection and recovered pen-tip position.
The resulting pen-tip trajectory is smoothed using a constant-velocity Kalman filter with innovation-based outlier rejection. Camera timestamps are retained throughout the pipeline to provide a common time axis for estimation and control.
Method
We transfer the classical online forward-Jacobian estimator from serial-arm visual servoing to a harder setting: a redundant, tendon-driven in-hand system. We make it work there through two additions: a deliberate excitation phase and nullspace grip stabilization.
Problem Formulation
For on-paper runs, the sheet is folded over itself or taped on the table such that the middle area of the paper bulges upwards by about a centimeter, absorbing the uncontrolled millimeter-scale vertical drift so that the pen tip remains in contact with the paper.
Locally, joint and task velocities are related by the task Jacobian, a matrix mapping joint velocities to task-space velocities.
Writing-Path Generation
After the excitation phase, the tracking phase begins, in which the writing-path generator sends the task command to the controller at each step. The trajectories are each reduced to a planar curve in the paper frame and traversed at constant tangential speed, so pen-tip speed is independent of curvature.
The letters A–Z are defined as cubic Bézier glyphs, while arbitrary single-stroke shapes within the writable range of the pen are also supported, including parametric shapes such as circles, flowers, hearts, and stars, as well as free-form SVG drawings.
The task-space command sent to the controller is a PID-plus-feedforward law on the tracking error, a proportional–integral–derivative command augmented by a velocity feedforward term.
The integral includes an anti-windup clamp, a limit on accumulated integral action, and the feedforward term equals the path tangent velocity at the current target.
Results
Each run establishes the grip, runs the excitation phase, and then tracks a reference path in air or on paper: a closed shape, a letter, or a free-form SVG drawing. Ablations change one component per run against the fully configured system.
Conclusion
We demonstrate that a rapid, embodied learning approach to dexterous manipulation can drive contact-rich in-hand writing on the anthropomorphic, tendon-driven ORCA hand by finger motion alone, after approximately 18 seconds of initialization, while continuing to adapt during writing.
Simulation experiments on two additional hand embodiments show the same formulation operating across substantially different hand kinematics. This fast, adaptive, and embodied approach could achieve a level of robotic writing accuracy on biomimetic hands not yet seen in approaches such as reinforcement learning or imitation learning that require extensive modeling or data-collection efforts.
These results suggest that a lack of a precise model or data is not necessarily what prevents real dexterity. The method’s speed and computational lightness further make it promising for applications where high-performance computing is unavailable, such as onboard prosthetics control.
Frequently Asked Questions
How does the system track the pen tip? A webcam tracks an ArUco marker attached to the pen and three markers defining the paper frame. The pen-tip position is recovered from a fixed marker-to-tip offset and smoothed with a constant-velocity Kalman filter.
How long is the initialization phase? The system requires approximately 18 seconds of initialization before the physical ORCA hand writes the alphabet.
What writing paths can the system follow? It supports cubic-Bézier letters A–Z, parametric single-stroke shapes, and free-form SVG drawings within the pen’s writable range.
Why is nullspace grip stabilization used? Nullspace grip stabilization helps maintain the pen grip while the redundant hand performs the writing task.