Force-Aware Trajectory Construction from Video and Audio
The method uses the first frame as the fixed starting point. Since the method is robot-centric, the initial frame must contain the robot end-effector.
For whiteboard wiping, carrot peeling, and lamp button pressing, every generated video is passed directly to segmentation without manual screening. For chocolate box stacking, 20 videos are generated and 10 are discarded because they deviate from the prompted grasp: the gripper grasps the box from its sides instead of sliding under it, which the parallel gripper cannot execute, or the end effector flips while holding the box. No further selection is applied.
Prompt-Conditioned Force Ordering in Generated Audio
To test whether the relative loudness of generated audio reflects the relative force magnitude specified in the task prompt, a controlled hammer-striking experiment is conducted. Starting from the same input image of a hand holding a hammer above a table, Seedance 2.0 is prompted with:
> “Strike the table exactly twice with a specified first force magnitude of force, then with a specified second force magnitude. Each strike must produce exactly one clean contact without bouncing,”
The generated audio preserves the specified force ordering in 18 of 20 trials, with 9 of 10 trials succeeding in each prompted order. These results suggest that, in this controlled impact setting, Seedance 2.0 can generate audio whose relative loudness reflects the force ordering specified in the prompt.
This tests whether the generator follows a prompted force order, not whether loudness measures physical contact force. These tests are shown in the supplementary video and project website.
Discussion and Conclusion
A pipeline was presented that jointly uses generated video and audio to construct force-aware trajectories for contact-rich manipulation. Across four real-robot tasks, the pipeline enables zero-shot execution where kinematic-only replay frequently fails.
Force-profile ablations show that the audio-shaped force profile matches or exceeds constant-force execution in success, with lower peak force and impulse. A controlled hammer-striking study shows that generated audio preserves the force ordering specified in the prompt. Finally, demonstrations collected with the pipeline train task-specific Diffusion Policies that generalize across object placements and appearances.
Limitations
SAM-Audio may retain unrelated sounds, making the resulting loudness profile an imperfect force proxy, although geometric contact gating, force bounds, and closed-loop regulation limit these errors.
The audio-shaped force profile is neither unique nor necessarily optimal, and a tuned constant force can perform comparably on sustained-contact tasks. Finally, the current pipeline requires time to generate the video and extract the trajectory and force profile. Consequently, the scene must remain unchanged during this process, and the system does not yet support real-time adaptation to environmental changes.
Future Work
Future work could incorporate real-time audio feedback during execution for closed-loop force adaptation and improve depth grounding in the generated video to reduce trajectory error.
Combining the pipeline with an AI agent could also close the loop: the robot could propose its own task prompts, execute the generated videos, and feed the outcomes and measured forces back into the next round of generation. In this way, force-aware task execution and data collection could become self-iterating.
Frequently Asked Questions
How are force-aware trajectories constructed? Generated video provides the trajectory, while generated audio is used to shape the force profile during contact-rich manipulation.
How well did generated audio preserve prompted force ordering? The generated audio preserved the specified force ordering in 18 of 20 hammer-striking trials.
Which tasks used generated videos without manual screening? Whiteboard wiping, carrot peeling, and lamp button pressing used every generated video directly for segmentation.
What is the main limitation of the current pipeline? Video generation and trajectory and force-profile extraction take time, so the scene must remain unchanged and real-time environmental adaptation is not yet supported.