P3-WAM combines three jobs in one model: reconstructing hand and camera motion from video, predicting what happens next, and planning actions toward a visual goal. The researchers report stronger results than evaluated baselines across these tasks, and demonstrate that generated human-hand motions can be transferred to a physical dexterous robot—a step toward systems that connect visual understanding with real manipulation.
What Did the Researchers Build?
P3-WAM is a unified world action model for dexterous manipulation: a system that predicts both how a scene will look and how hands and the camera will move. It takes egocentric video—footage from the viewpoint of a person performing a task—along with a text instruction. For planning, it can also use a goal image showing the intended final arrangement.
The model supports three modes. Perceive estimates hand and camera motion from observed video. Predict uses observed actions to forecast future video and motion. Plan generates a possible action sequence and the corresponding video for reaching a goal. In each mode, it produces explicit hand and camera trajectories alongside visual output, rather than treating the future video as the only result.
A shared model matters because these tasks overlap. Reconstructing hand motion can help the system learn how actions relate to visual changes, while predicting those changes can provide useful context for planning. P3-WAM is built around a pretrained video-generation model, extended with a pathway for structured hand and camera actions. The paper also describes demonstrations in which its human-hand motion is converted into trajectories for a dexterous robot.
What Results Did P3-WAM Report?
The researchers report that unified training improves performance across all three modes compared with training modes separately. P3-WAM outperformed the evaluated baselines on the Perceive and Predict tasks. For Plan, its generated videos received higher vision-language model scores for the quality of the action process and adherence to the instruction. The supplied paper text does not include the numerical benchmark scores, so the size of those gains cannot be compared here.
Perception was evaluated on four datasets: ARCTIC, HOT3D, EgoLive, and EgoDex-MANO. The evaluation includes presence-aware 3D hand-joint errors, which account for whether a hand is visible, as well as measurements in camera-space, root-relative, and aligned coordinates. Prediction tests include video quality and hand-region comparisons; the paper describes metrics including Fréchet Video Distance (FVD) and intersection over union (IoU).
The planning evaluation included 17 task categories. For a human-rating study, each model produced 188 videos; each video contained 81 frames at 16 frames per second. Reviewers scored five dimensions on a 1–5 scale, while a separate vision-language-model evaluation scored generated videos. The provided text describes the evaluation setup but not aggregate human-rating results.
Physical tests add a different kind of evidence: two generated manipulation sequences were converted into robot trajectories and replayed in real environments. The paper reports demonstrations, but the supplied text does not give numerical success rates.
How Does P3-WAM Work?
P3-WAM builds on a pretrained video-generation backbone. Video frames are represented internally as compressed visual data, while observed hand and camera states are encoded as compact action tokens. A frozen action encoder produces three tokens per frame: one for the camera, one for the left hand, and one for the right hand. Hand skeletons and camera-ray maps also provide spatial cues that align actions with the image.
A mixture-of-transformers architecture lets video and action information interact through shared attention. In practical terms, the model can use visual context when estimating an action, and action context when predicting how the scene will change. During training, randomized modality visibility exposes the model to different combinations of observed and missing video or action information. That helps support the three operating modes within one framework.
The model handles video and actions differently at generation time. In Perceive, it reads the observed video and predicts the full action sequence in a single pass. In Predict and Plan, it iteratively generates missing video frames while keeping observed frames fixed; it predicts the action trajectory alongside that process. A decoder then turns the generated video and action representations into visible frames, hand motion, and camera motion. Hand motion is represented using MANO, a model of hand shape and joint articulation.
For Plan, the goal image supplies spatial information that a text instruction might leave unclear—for example, exactly how far apart two blocks should end up. Longer tasks can be broken into stages, with a visual subgoal for each planning window. The model then generates the video and motion connecting the current observation to that window’s target.
Why Does This Matter for Robotics?
Robots that handle small objects need more than a plausible-looking video or a sequence of hand poses. They need actions tied to what is visible, an estimate of how the scene will change, and a way to express the intended result. P3-WAM connects those pieces in one model, which could make it easier to move from observing a manipulation to anticipating or planning one.
Its use of visual goals is especially relevant for tasks where verbal instructions are underspecified. “Separate the blocks” does not state the exact spacing or arrangement; an image can. Breaking longer tasks into visual stages also offers a way to represent intermediate targets, rather than asking a planner to jump directly from the initial scene to the end state.
The robot demonstrations show a possible path from generated human motion to a robot hand: reconstruct objects and scene motion from the video, retarget the hand movement, and replay the resulting trajectory. This is not yet a complete, feedback-driven robot controller, but it tests whether generated behavior can be grounded in physical execution. Teams exploring dexterous platforms can browse humanoid robots on Robot Overflow.
What Are the Limitations and Open Questions?
Monocular video does not provide a reliable absolute depth measurement, so estimating a hand’s global 3D position is inherently ambiguous. The paper identifies depth or tactile sensing as potential ways to add stronger geometric information. Those sensors could help resolve uncertainty about hand-object contact and distance.
The physical robot tests use open-loop execution: the robot replays a planned trajectory without adjusting it in response to what happens during execution. That leaves the system vulnerable to small errors that accumulate, unexpected object movement, or imperfect contact. Closed-loop feedback would allow corrections while the task is underway.
The demonstrations cover two real-world experiments, not a broad range of objects, environments, or robot designs. The supplied text also does not provide numerical benchmark gains or robot success rates, limiting assessment of the size and reliability of the reported improvements. Further testing would need to establish how well the approach transfers across varied tasks and hardware.
What Questions Do Readers Often Ask?
Does P3-WAM directly control a robot? It predicts human-hand and camera motion; a separate reconstruction and retargeting process converts the generated motion into a robot trajectory.
What does the model generate in Plan mode? It generates a predicted video and an explicit hand-and-camera trajectory, conditioned on an instruction and visual goal.
Does planning require a goal image? The default Plan setup uses a goal image, which specifies the desired visual arrangement. The paper also evaluates a version without that image.
Has the model been tested on a physical robot? Yes. The researchers describe two real-world executions, using open-loop replay of trajectories transferred from generated human-hand motion.
What Is the Bottom Line?
P3-WAM brings hand and camera motion, video prediction, and goal-directed planning into one model, with reported improvements over the evaluated baselines. Its robot demonstrations are promising, but broader physical testing and closed-loop control are important next steps.