Connecting an agent's actions to their effects in the world, and vice versa, is fundamental to purposeful interaction. In sensorimotor control, skilled behavior couples a forward model that anticipates sensory consequences of movement with an inverse model that recovers movement needed to realize a desired state. Robotic world models should similarly support both directions of reasoning in a single predictive framework.
Introduction
This work introduces Masked Visual Actions, a pixel-space control interface for pretrained video models, together with an efficient adaptation recipe based on masked examples from real and simulated data. Forward and inverse robot world-modeling problems can be cast as complementary conditional prediction problems of the same video model, obtained by revealing different entities in the scene. This framework is validated in both simulation and the real world across three applications in robot manipulation: policy evaluation, model-based planning, and inverse modeling.
Dataset Construction
The masked modeling dataset is constructed by combining real-world videos from DROID and simulation data from Robocasa. Both success and failure trajectories from both datasets are used. Two approaches are followed to construct masked conditioning for each video, based on video segmentation and rendering the robot state.
Model Implementation and Training
Wan-Fun-Control 2.2 14B is used as the base model. The masked conditioning video is encoded using the same autoencoder as the video model, with concatenation as the conditioning mechanism. Concatenation is appropriate as the conditioning signal is spatially aligned with the desired output video. For the missing region from the masked conditioning, it is set to a uniform gray background. Instead of finetuning the entire model, LoRA finetuning is used with rank 256, and a batch size of 4 using 8 NVIDIA H200 GPUs. The model is trained for approximately 10,000 steps over 4 days.
Experiments
Masked Visual Actions is evaluated as a control signal for world modeling, assessing visual fidelity and controllability against prior work while highlighting generalization to embodiments unseen during training. Diverse robotic applications of the video model are evaluated, including planning by evaluating sampled trajectories, policy evaluation, and using the video model as the policy itself through inverse modeling.
Robotics Applications
Multiple applications of the unified world model in robotics are highlighted. The model is used as a forward model to simulate robot actions, demonstrating its use for planning and policy evaluation. The model is also used as an inverse model: given the desired object motion as a masked visual action, a video of the robot performing the desired object manipulation is generated, and actions are extracted using a learned inverse dynamics model. Robocasa serves as the simulation environment across these applications.
Simulation Policy Training for Planning and Policy Evaluation
For the simulation policies, they are evaluated against the success conditions provided by the benchmark. In planning, these policies serve as proposal generators: action candidates are sampled from the policy, each candidate is simulated with the video model, and the predicted futures are ranked with the VLM-based evaluator. In policy evaluation, the same class of policies provides the action trajectories whose success rate is compared between the simulator and the video model.
Learned Inverse Dynamics for Action Extraction
Diffusion Policy, ACT, and SmolVLA baselines are included in this experiment, but only as narrow controls for the action-extraction setting. Those baselines operate on the first frame of the generated video rather than the full synthesized interaction, and therefore do not require a separate dedicated discussion outside this subsection.
Hardware Setup
The sensing setup consists of two ZED Mini 2 cameras. One camera is mounted on the robot's wrist and provides an egocentric view near the end-effector. The second camera is mounted externally and provides a fixed third-person view of the workspace. Both cameras record RGB frames at 1280 × 720 resolution and 30 Hz.
Camera and Robot Frame Calibration
The external camera is calibrated by observing the same table-mounted AprilTag. Since the wrist-camera pose is known from the wrist-camera calibration, the external-camera pose relative to the robot base can be computed from the detected tag pose. The table-mounted AprilTag provides a shared reference frame between the robot, wrist camera, and external camera. Calibration quality is verified by projecting the robot model into the camera images and checking their alignment with the observed frames.
Trajectory Representation
Each rollout is stored as a trajectory where o_t and o_t_wrist are the external and wrist RGB observations, and q_t is the measured robot joint state. All data are stored in HDF5 format. Each file corresponds to one rollout and contains RGB frames, joint states, timestamps, and calibration metadata. For experiments involving visual robot masks, the stored joint states are combined with the Franka kinematic model, custom end-effector geometry, and calibrated camera parameters to render the robot geometry into the image plane.
Discussion and Conclusions
By finetuning a pretrained video model on a small amount of Masked Visual Actions data, the prior of the video model is efficiently leveraged to synthesize counterfactuals by conditioning on a subset of scene entities. The model can simulate robot actions when conditioned on robotic embodiment visualization as an action-conditioned forward model, and when acting as an inverse model, it synthesizes suitable robot motion to realistically manipulate the object.
Limitations
The model, similarly to existing generative models, learns the correlation between object interaction rather than causal relationships, which remains an open research question. The method is naturally limited by the base video model's capabilities, in terms of both inference speed and what it can express, as it re-purposes the model's prior rather than modifying its capabilities.
Societal and Broader Impact
By enabling video-based policy evaluation, planning, and inverse modeling, this work could lower the cost of developing robotic systems and make robot learning more accessible. However, the same capabilities could also be used for unsafe or unauthorized robotic behaviors, highlighting the need for responsible use and deployment.
Frequently Asked Questions
What is Masked Visual Actions? Masked Visual Actions is a pixel-space control interface for pretrained video models that allows robots to learn both forward and inverse world models from masked video examples.
How is the model trained? The model uses LoRA finetuning on a Wan-Fun-Control 14B base model, trained for 10,000 steps on 8 NVIDIA H200 GPUs using data from DROID and Robocasa datasets.
What robotics applications does this enable? The model enables policy evaluation, model-based planning, and inverse modeling where the video model itself generates robot actions based on desired object motion.
What are the key limitations? The model learns correlations rather than causal relationships, and its performance is bounded by the base video model's capabilities in terms of speed and expressiveness.