We present ContactMimic, a framework that augments humanoid keypoint tracking with an explicit, test-time controllable contact label. By training a single policy on paired motions that share keypoint structure but differ in contact commands, and supervising it with contact-aware rewards, the policy learns to make or suppress physical contact on command. Across diverse human-object interaction motions in simulation and on the real Unitree G1, the policy follows the commanded contact, outperforms a keypoint-only baseline on contact metrics at comparable tracking accuracy, and even encodes its runtime contact state from proprioception alone.
Introduction
For the second question, we use contact data for policy training in two ways. First, we inject contact commands into the policy. Second, we include rewards that encourage the policy to match the commanded contact patterns: a contact label matching reward that measures the agreement between the reference and actual body-part object-part contacts, and a contact distance reward that guides relevant body parts toward (or away from) the object surface based on the reference label.
Generating Motion Pairs to Break Correlations Between Keypoints and Contacts
Our proposed contact and keypoint tracker needs reference trajectories with per time step keypoint and contact labels for training. We extract such contact label-paired keypoint trajectories from human MoCap data. Existing works only extract keypoints, we extend them to also extract contact information. As our experiments show, just training on this data alone doesn't lead to good contact control due to the correlations between keypoint patterns and contacts. Thus, to achieve genuine contact control, we propose augmentation strategies that break this correlation.
Augmented trajectory. For each retargeted clip, we synthesize several augmentations that preserve the overall motion structure but break the correlations between the keypoint positions and the contact labels for the task relevant contacts (e.g. wrist-box contact for the task of lifting a box).
Experimental Setup
We use the Unitree G1 humanoid robot (29 DoF) in Isaac Lab with PhysX rigid-body simulation. Each environment contains one interaction object, which is either fixed or free depending on the motion.
Motion Dataset. We train on 10 motion clips from the HUMOTO dataset, spanning a range of human-object interaction categories.
Experiments
Our experiments across diverse human-object interaction motions seek to answer 5 key questions: Do our policies follow the commanded contact conditioning? Does contact control transfer to the real world? Is keypoint control alone sufficient, or does a state-of-the-art keypoint-tracking baseline already produce accurate physical contact? Is our data engineering strategy necessary for learning contact control? To what extent does the policy's internal representation already encode the robot's actual runtime contact state, given only proprioceptive observations?
Policy Representations Encode Runtime Contact State
As most robots do not have whole-body contact sensors, we made the deliberate decision to not require the actual runtime contact state for the different robot body parts as a policy input. Partly our rationale was that the contact state could be inferred using proprioception. We assess this quantitatively by computing the contact state prediction accuracy of a linear probe on the policy inputs and the policy's intermediate representations.
Results. For both the raw observation and Layer 2 representations, the F1 scores are very high, and much higher than the chance performance (i.e. the number of time steps with contact) as well as the reference label itself. Thus, despite receiving no explicit contact sensing at inference time, the policy has a sense of the robot's runtime contact state and can adapt its behavior to fulfill the commanded contact state.
Discussion and Limitations
We presented ContactMimic, a framework that augments humanoid keypoint tracking with an explicit, test-time controllable contact label. By training a single policy on paired motions that share keypoint structure but differ in contact commands, and supervising it with contact-aware rewards, the policy learns to make or suppress physical contact on command. Across diverse human-object interaction motions in simulation and on the real Unitree G1, the policy follows the commanded contact, outperforms a keypoint-only baseline on contact metrics at comparable tracking accuracy, and even encodes its runtime contact state from proprioception alone. We believe explicit contact conditioning is a simple and general interface for expressing contact-rich tasks, and a step toward humanoids that interact with the world purposefully rather than incidentally.
Our approach has several limitations. First, we train a separate policy per motion rather than a single universal contact-conditioned tracker. Jointly training a contact-conditioned policy across motions would be a natural next step. Second, we built our approach on high-quality human-object interaction data from the HUMOTO dataset which limits the diversity of interactions we can currently cover. Developing methods that can use in-the-wild video data instead can help. Finally, our real-world evaluation spans five motions on a single robot; broader hardware validation is future work.
Frequently Asked Questions
How does ContactMimic control contact without physical contact sensors? The policy learns to infer its runtime contact state from proprioceptive observations alone, achieving high F1 scores when predicting contact labels from its internal representations.
What augmentation strategy breaks the correlation between keypoints and contacts? For each retargeted motion clip, multiple augmentations are synthesized that preserve the overall motion structure but alter the contact labels for task-relevant body parts and object surfaces.
What robot hardware was used for evaluation? The Unitree G1 humanoid robot (29 degrees of freedom) was used in both Isaac Lab simulation and real-world experiments.
Does ContactMimic require one policy per motion or a single universal policy? Currently a separate policy is trained per motion; training a single universal contact-conditioned tracker across all motions is identified as future work.