Open-world manipulation requires rapid responses to contact and slip events. When a fingertip first makes contact, or when a grasp transitions from sticking to sliding, the relevant hand-object interaction cues are often brief, subtle, or visually occluded. We present VibeAct, a framework that estimates these contact and slip interactions using piezoelectric microphones embedded in a robot hand and leverages them as tactile observations for learned dexterous manipulation policies.
Learning dexterous manipulation directly from real-world vibrotactile observations is equally impractical. Collecting large-scale contact-rich demonstrations is labor-intensive, while online reinforcement learning is prohibitively expensive and potentially unsafe due to unstable exploration during early training. Together, these constraints motivate a representation that can bridge real vibrotactile sensing and simulation-based control.
The central insight of this work is that contact and slip provide an interface between sensing and control for sim-to-real policy learning. Contact onset, binary slip, and scalar slip magnitude are low-dimensional and task-relevant quantities that can both be estimated from real microphone signals and computed directly in a contact simulator. Rather than training policies on raw audio, VibeAct learns a tactile estimator that maps microphone waveforms to this physical representation, and trains reinforcement learning policies in simulation using the same representation as an observation channel. The estimator solves the sensing problem, while the simulator solves the control problem.
We instantiate VibeAct on a dexterous hand mounted to a robot arm, with piezoelectric microphones embedded in each fingertip. We replay real-world recordings of robot trajectories and object poses in a calibrated digital clone, and use the simulator's contact solver to generate contact and slip labels for training the tactile estimator. Using the same tactile representation alongside proprioceptive and point-cloud observations, we train PPO policies entirely in simulation. We then evaluate the learned policies on contact-rich manipulation tasks, including in-hand reorientation and peg insertion.
This paper makes three contributions: (i) a sim-to-real framework for vibrotactile dexterity based on a shared contact-and-slip representation that bridges real vibrotactile sensing and simulation-based policy learning; (ii) a digital-clone data labeling pipeline that automatically generates per-finger contact and slip supervision from real-world demonstrations; and (iii) an empirical study showing that this representation serves as an effective tactile observation for reinforcement learning across contact-rich dexterous manipulation tasks.
Method
VibeAct has three components. First, a teleoperation pipeline records robot and object states, synchronized with vibro-acoustic signals, then replays trajectories in a calibrated digital clone in simulation to generate per-finger contact and slip labels as a tactile representation. Second, a tactile estimator learns to map real microphone signals to this tactile representation. Third, RL policies are trained in simulation using proprioception, point clouds, and the same tactile representation as observations. At deployment, the trained estimator replaces the simulator-derived tactile channel.
Tactile Representation
The representation intentionally excludes privileged simulator information such as contact location, surface normal, tangential direction, object identity, and force vectors. This restriction is important because the representation must remain predictable from the physical microphone signals and transferable across objects and tasks. At the same time, it preserves the key feedback required for reactive manipulation: whether a fingertip has just touched the object, whether slip is occurring, and the severity of that slip.
Tactile Estimator
We assume that contact onset and slip retain the same physical meaning in both simulation and real-world vibro-acoustic sensing. Under this assumption, we train a tactile estimator that maps real microphone audio to the contact and slip labels generated by the digital-clone replay pipeline.
Policy Learning
We train consistent policy architectures across all five tasks in MuJoCo simulation with PPO, varying only the scene and task reward. Each task instantiates the same xArm7 and LEAP hand model under per-episode domain randomization of object friction, mass, scale, and pose, so the policy must rely on contact feedback rather than memorized geometry.
Tactile Estimator Evaluation
We collect two teleoperated datasets to train and evaluate the tactile estimator before policy deployment. The fixed-object dataset (five hours) contains interactions with objects rigidly mounted to the table, inducing slip purely through robot motion. This scales diverse contact and sliding events but yields slip dynamics unlike dexterous manipulation. The moving-object dataset (under two hours) involves in-hand manipulation of free-moving objects, with slip arising from finger motion, arm motion, and gravity. We pretrain on the fixed-object data to learn general vibro-acoustic patterns, then fine-tune on the moving-object data to match deployment. We evaluate all models on a held-out moving-object split, reporting F1 for binary contact-onset and slip-presence detection and MAE for slip magnitude, averaged over the four fingers.
We further evaluate parameter sharing across fingers. Sharing the encoder while keeping independent prediction heads reduces contact-onset F1 by 2.3% and raises slip-magnitude MAE by 1.9% relative to VibeAct. Sharing both the encoder and prediction heads further compounds the degradation, reducing contact-onset F1 by 3.7% and increasing MAE by 4.8%. These results suggest that each fingertip exhibits distinct contact geometry and vibration propagation characteristics that benefit from dedicated feature extraction and prediction heads.
Task Performance
We evaluate VibeAct against a proprioception and point cloud baseline (Prop+PC) with no tactile input, and ablate the tactile representation by incrementally adding contact onset, slip presence, and slip magnitude channels. We report the results based on 100 trials trained across 3 random seeds.
The ablations reveal that slip magnitude is a critical channel where adding it alone drives the largest jump in performance across all tasks. Contact onset gives inconsistent gains in isolation and in some tasks hurts performance, likely because sparse onset pulses alone carry limited information for sustained reactive control, while slip presence offers intermediate benefit. Together, these results indicate that the continuous slip magnitude signal is the primary source of VibeAct's advantage, providing the graded feedback needed for contact-rich manipulation.
We deploy the simulation-trained policies directly on the physical platform, replacing the simulator-derived tactile channel with the trained estimator. VibeAct improves over Prop+PC on every deployed task. These results indicate that the contact-and-slip representation, learned entirely from real microphone recordings and used as a drop-in observation channel, retains its benefit under real sensing and actuation noise.
Limitations
VibeAct's compact contact and slip representation discards information in the raw vibration signal, such as contact location, surface texture, and interaction dynamics beyond slip magnitude; richer representations could help but would demand closer sim-to-real acoustic alignment. The tactile estimator is also tied to a fixed hardware configuration and may need recalibration if microphone placement, finger material, or object properties change. The digital-clone labeling, while annotation-free, depends on accurate object pose tracking, limiting use in unstructured settings. Finally, the policy treats the tactile representation as a flat vector, leaving room for architectures that model the spatial and temporal structure of per-finger contact events.
Network Architecture
The same network architecture is used for every task and every tactile variant, where only the action head's output dimension changes with the task action space.
Reward Functions
The shaped reward combines five terms:
- Climb progress: Reward for new positive increments of the held-object's world-z height (one-sided, ratcheting on best-so-far)
- Per-finger advance: Reward per fingertip for new descent on the object body axis (one-sided)
- Hold: Reward per step, proportional to the number of fingertips in contact
- Drop: One-shot penalty if the held object falls more than a threshold below its initial pose
Frequently Asked Questions
What sensing hardware does VibeAct use? VibeAct uses piezoelectric microphones embedded in each fingertip of a dexterous robot hand to capture vibro-acoustic signals during contact and slip events.
How does VibeAct bridge simulation and reality for tactile sensing? It trains a tactile estimator that maps real microphone signals to a shared contact-and-slip representation, which is also used as the observation channel for reinforcement learning policies trained entirely in simulation.
What tasks were used to evaluate VibeAct? The framework was evaluated on contact-rich manipulation tasks including in-hand reorientation and peg insertion using a dexterous hand mounted to a robot arm.
What is the most critical tactile signal for VibeAct's performance? Slip magnitude is the most critical channel, as it provides graded continuous feedback that drives the largest performance gains across all tasks tested.