Correcting millimeter residuals requires a policy both to perceive them and to resolve them with actions. Without error channels scaled to the tolerance, the arm settles at a biased equilibrium short of the goal. Input normalization scales each channel by its spread over the workspace, so millimeter differences occupy a negligible fraction of the joint, goal, and keypoint inputs. PolicyIK therefore represents pose error at two scales and adapts its step size to the error.
Multiscale Pose Feedback and Action Scaling
Following keypoint-based pose representations, three noncollinear local offsets [formula not provided in the source] define pose errors as position differences, expressing translation and orientation errors.
To let the policy perceive millimeter errors, the error is also expressed in the current end-effector frame and clipped at a coarse and a fine scale.
Reward Design
The reward combines terms for the approach and for the terminal phase.
Tracking guides the approach; progress and dwell reward reaching and holding the tolerance; and costs regularize motion. Appendix B lists all terms and weights.
Simulation Training and Deployment
PolicyIK is trained with PPO and generalized advantage estimation in 4,096 environments. Training rollouts last 12 seconds and resample the goal every 4 seconds without resetting the arm. Each rollout therefore chains three attempts, and later attempts start wherever the previous one left the arm. Appendix B gives the architecture and training settings.
A strict failure in a single attempt ends either near the goal with a small residual error or farther from it. When further attempts are allowed, a fixed protocol handles each case. A failure within 1 cm of the goal is retried with an error-corrected goal that cancels the residual. Other failures receive up to eight attempts that first command a transient goal to move the arm to a different configuration. Every attempt is judged against the original goal.
Real-Robot Training
Large-rotation goals often require a detour in joint space, during which the pose error first grows. A standard critic tends to assign similar values to poses that look close, so it favors moving straight toward the goal. Following LAN, the critic embeds the state and action with one network and the goal with another, then predicts the value from their difference. This architectural design has been shown to better capture the temporal distance between observations and goals.
Experiments
The experiments evaluate PolicyIK’s precision in simulation, the contribution of each component, and its use as a low-level controller in simulated pick-and-place and on a physical arm.
Training Tasks
Each training episode lasts 150 control steps with one goal. Its task comes from one of two sources.
Environment and Reward
Equation 2 clips the reference at the joint bounds. A joint that holds a goal near its bound against gravity needs a reference beyond the goal configuration, which a hard clip at the bound forbids. On hardware, the clip is relaxed by an unspecified margin while the measured joint is still inside the bound. Once the measured joint reaches the bound, no further outward step is accepted, so the joint itself never exceeds it.
Frequently Asked Questions
Why does PolicyIK represent pose error at two scales? Input normalization can make millimeter differences negligible compared with the spread of inputs over the workspace. The two-scale representation helps the policy perceive and resolve these small errors.
How is PolicyIK trained in simulation? It uses PPO and generalized advantage estimation in 4,096 environments. Each 12-second rollout resamples the goal every 4 seconds without resetting the arm.
How does the retry protocol handle failed attempts? A failure within 1 cm of the goal is retried using an error-corrected goal that cancels the residual. Other failures can receive up to eight attempts, beginning with a transient goal that moves the arm to a different configuration.
Why does the real-robot critic use separate state-action and goal embeddings? Large rotations may require a joint-space detour that initially increases pose error. Predicting value from the difference between the state-action embedding and goal embedding is intended to better capture temporal distance to the goal.