NavWAM learns a joint world-action prediction model that turns visual foresight into a closed-loop policy for goal-conditioned visual navigation. By representing future observations, goal-progress values, and executable action chunks in a shared latent sequence, NavWAM learns future perception, progress estimation, and action generation as one coupled prediction problem.
Visual Foresight and Action Selection
Under partial observability, successful navigation requires two coupled abilities: predicting how the view will change after moving, and choosing an action that brings the robot closer to the goal. Direct policies learn this as a single action-prediction problem, which is efficient at test time, but the model is not explicitly trained to predict what it will see after executing the action.
NWMs make future prediction explicit by predicting future observations conditioned on candidate actions, however they still require a separate action-selection procedure to choose which candidate action to execute. This creates a gap between direct action prediction and explicit future prediction.
From Navigation World Models to NavWAM
The NWM approach shows that future egocentric image prediction can support navigation by using CEM-based planning to sample candidate action sequences, predict their future observations, and select the best-scoring trajectory. However, this pipeline still treats future prediction and action selection as separate steps: the model predicts possible futures, while an external procedure decides which future indicates goal progress and which action should be executed. This separation increases test-time computation and makes closed-loop behavior depend on candidate sampling, scoring, and optimization.
NavWAM addresses this separation by learning a joint world-action prediction.
Method
NavWAM parameterizes a closed-loop navigation policy using a pretrained video world model. Rather than using the model only to predict future views for externally planned actions, NavWAM represents the variables needed for navigation control as a shared latent canvas. The current observation, image goal, robot state, action chunk, future egocentric observations, and goal-progress value are assigned to different latent frames in the same diffusion-transformer sequence. This turns goal-conditioned navigation into a joint denoising problem over future perception, progress estimation, and executable motion.
World-Action Latent Canvas
Image frames are encoded as standard video latents through the causal VAE. Non-image variables, including states, action chunks, and the scalar value, are normalized and broadcast over the latent spatial grid. Their predictions are recovered by averaging the denoised entries of the corresponding frame. This preserves the pretrained video-transformer interface while allowing one model to jointly predict visual and non-visual navigation variables. Although the same canvas could in principle be extended to language- or object-specified goals through the goal frame or text-conditioning interface, we focus on image-goal navigation in this work.
Training Objective
The denoising loss is applied to the generated frames of the latent canvas. The loss includes a diffusion weighting term and is conditioned on observed-frame masks, conditioning embeddings, and observed latent frames. We upweight the action frame relative to the other prediction frames so that the low-dimensional action signal is not dominated by the high-dimensional image reconstruction loss.
NavWAM does not introduce separate action or value heads. Instead, actions, states, and scalar values are decoded from their corresponding denoised latent frames. Thus, action generation, future-state prediction, future-view prediction, and value estimation are trained as parts of the same world-action denoising objective, rather than as auxiliary losses added to an action-only policy. For navigation, the value frame is trained to represent goal progress rather than a generic reward-to-go. This encourages the model to estimate whether the predicted future state moves the robot closer to the specified goal.
Inference
NavWAM also supports optional value-guided best-of-N sampling. In this mode, the model draws N candidate action chunks, evaluates their predicted futures and goal-progress values using the auxiliary world-model and value modes, and executes the chunk with the highest predicted value. This optional mode is not used in the reported main results. All reported main results use the default policy mode without CEM-style action search.
Robot Platform
The base is a Direct Drive Tech Diablo fitted with a 3D-printed frame that holds an Intel RealSense D455 RGB-D camera, a Livox Mid-360 LiDAR, an NVIDIA Jetson AGX Orin, and a dedicated battery for the Orin and the LiDAR. The platform is controlled through ROS 2 via standard cmd_vel commands, which are used to execute the local-frame action chunks predicted by NavWAM during deployment.
Training Curriculum
NavWAM is trained in three phases. Each phase initializes its weights from the previous phase and writes a single checkpoint used throughout the main paper. Training uses the success-only split of hm3d simulator trajectories generated by the navigation policy, with rollout demonstrations sampled alongside expert trajectories. The hm3d phase uses a long chunk to expose the model to long-horizon visual foresight from the simulator.
Training Hyperparameters
The diffusion objective upweights the action frame relative to the future-image frames by a multiplier. The model is trained under three conditioning modes: policy mode conditions on observed frames and predicts action, future state, future images, and value; world-model mode also conditions on the action frame; value mode conditions on all frames except the value scalar itself.
Conclusion
NavWAM learns future perception, progress estimation, and action generation as one coupled prediction problem. In evaluations, this joint formulation improves navigation performance over planning-based world-model baselines without CEM-style action search in the default policy mode, remains competitive with a larger direct navigation policy, and transfers to closed-loop real-robot image-goal navigation. These results suggest that future prediction is most useful for robot navigation when it is learned together with the action and value targets that determine closed-loop behavior, rather than treated as a separate planning module.
Evaluation focuses on image-goal navigation in indoor environments, and broader evaluation on language- or object-specified goals remains future work. The real-world study is limited in scale, covering 24 closed-loop episodes across four environments.
Frequently Asked Questions
What makes NavWAM different from traditional navigation world models? NavWAM learns a joint world-action prediction that combines future observation prediction, action generation, and value estimation into a single diffusion model, rather than treating future prediction and action selection as separate steps.
How does the latent canvas work? Non-image variables like robot state, action chunks, and goal-progress values are normalized and broadcast over the latent spatial grid and decoded from their corresponding denoised latent frames, preserving the pretrained video-transformer interface.
Does NavWAM require CEM-style action search at test time? No, the default policy mode uses a single denoising pass without CEM-style action search, though an optional value-guided best-of-N sampling mode is available.
What are the main limitations of NavWAM? The evaluation focuses on image-goal navigation in indoor environments, the real-world study is limited to 24 episodes on one platform, and the inference cost is dominated by diffusion chain length.