Manipulation policies are predominantly evaluated by binary task success. Success aggregates perception, grasping, execution, and strategy into a single bit, so policies can fail for different reasons yet receive the same score. It can also mislead: a policy may receive a passing score even when its commanded gesture is wrong. For example, a behavior-cloning policy trained on doors opens held-out drawers in more than half of its episodes using the arcing gesture typical of doors—a rotation the drawer’s joint does not admit—yet each episode is scored as a success. A good evaluation should measure the commanded gesture against the target’s kinematics.
Introduction
Manipulation policies are evaluated predominantly by binary task success. Success aggregates perception, grasping, execution, and strategy into a single bit, so two policies can fail for different reasons and obtain the same score. It can also mislead, reporting a healthy-looking score while the commanded gesture is wrong.
For example, a behavior-cloning policy trained on doors opens held-out drawers in more than half of its episodes with the arcing gesture typical of doors. This rotation is not admitted by the drawer’s joint, but success scores each episode as a pass. A good evaluation should measure the commanded gesture against the target’s kinematics.
Constraint-Referenced Strategy
Constraint-Referenced Strategy (CRS) isolates the commanded gesture from confounds by evaluating a counterfactual: assuming the handle is already grasped at a valid pose, is the policy’s commanded gesture kinematically consistent with the object’s constraint?
CRS answers this through teacher-forced probing. A scripted controller, the oracle, places the robot at reference states along an opening sequence, and the policy’s commanded end-effector twist is recorded at each state. The reference states determine where the policy is evaluated, while the joint geometry determines what constitutes a valid command.
Frequently Asked Questions
What does binary task success fail to capture? It aggregates perception, grasping, execution, and strategy into one score, so different failures can receive the same result.
How can a success score be misleading? A policy can receive a passing score even when its commanded gesture is inconsistent with the object’s joint constraint.
What does CRS evaluate? It evaluates whether a policy’s commanded gesture is kinematically consistent with an object’s constraint, assuming a valid grasp.
How does CRS probe a policy? An oracle places the robot at reference states along an opening sequence, and the policy’s commanded end-effector twist is recorded at each state.