Reasoner
We retain the autoregressive reasoner of Cosmos3 without architectural modifications. The reasoner processes multimodal observations and language context to support embodied understanding, spatial grounding, and task planning. For visual planning, it takes the task goal, initial and current observations, and preceding subtask history as input, and predicts the next executable subtask together with a detailed description of its anticipated visual evolution and an estimated duration. These outputs provide the generator with structured textual conditions specifying the intended interaction and its temporal extent. During multi-step rollout, the final frame of each generated clip becomes the updated observation for subsequent reasoning, connecting subtask planning with the visual consequences of preceding predictions.
Visual Tokenization and Input Organization
For video generation, we encode visual inputs using a causal video VAE with spatial and temporal compression factors of 16 and 4, respectively. The resulting latents are spatially patchified into visual tokens. Both scene-memory frames and the main video follow this VAE pathway and are processed by the generator branch.
Each scene-memory frame is encoded independently, producing a group of visual tokens that represents its spatial content. Four reference frames therefore correspond to four groups of tokens, whose individual lengths depend on the frame resolutions. When scene memory is enabled, these groups are placed after the reasoner’s textual context and before the main video tokens.
At each denoising step, the generator processes the memory and main video tokens through its existing attention layers while attending to the reasoner’s textual representations. The main video can therefore access the scene references directly, without introducing a separate memory encoder or additional cross-attention modules.
Cross-Chunk Scene Memory
Long-horizon planning requires the model to retain visual information beyond the current observation, particularly when objects or scene regions leave the field of view. We maintain four reference frames comprising a persistent initial observation, two keyframes selected across earlier history, and a reference associated with the most recent step. This selection preserves both broader scene coverage and recent task context. When the history is insufficient, additional frames are selected from available historical clips; at the first step, the initial observation is repeated to fill the reference slots. During rollout, the initial observation remains fixed while the historical references are updated from generated videos.
We complement these sparse references with a continuous video prefix from the preceding chunk. Unlike the independently encoded memory frames, this prefix occupies the beginning of the main video’s temporal sequence. The preceding tail and the current segment form a continuous sequence, with the shared boundary frame included only once. The reference frames serve as visual anchors for scene appearance and layout, while the temporal prefix preserves recent motion and interaction context. Together, they form cross-chunk scene memory for successive visual predictions.
Generation Modes and Fixed Conditions
We support three generation modes through different arrangements of visual conditions within the main video sequence. In image-to-video (I2V) generation, the first latent frame is fixed to the encoded starting observation, and the model predicts the subsequent video. In first-and-last-frame-to-video (FL2V) generation, both the first and final latent frames are fixed, providing boundary conditions for generating the intervening transition. In cross-chunk video continuation, the initial latent frames are fixed to a continuous prefix from the preceding video’s tail, allowing the model to predict the next segment from recent motion and interaction context. During multi-step rollout, this prefix is extracted from the previously generated video.
Across these modes, the scene-memory latents and the conditioning latents within the main video remain fixed throughout denoising. The generator updates the remaining video positions while attending to the textual context and visual conditions. During training, only the positions to be predicted contribute to the generation loss. The same conditioning interface accommodates observed and self-generated visual inputs, supporting the self-forcing strategies used during generator fine-tuning.
Data
The paper indicates that the data used to fine-tune the reasoner and generator are summarized in Table 3.1; the table data are not included in the provided text.
General Understanding Data
To maintain broad embodied understanding and reasoning capabilities, we use a 10% subset of the RynnBrain pretraining corpus as the foundation of our understanding data. Reasoning about embodied tasks requires the model to interpret task intentions, recognize objects and their properties, understand spatial relationships, and track changes across successive observations. These capabilities are particularly important for long-horizon visual planning, where each subtask must be grounded in the current scene and considered in relation to subsequent operations. They also support the interaction between understanding and video generation: the model needs to identify which aspects of the scene are relevant to a task, formulate the corresponding visual prediction, and interpret the predicted changes to inform further reasoning.
The data cover general image and video understanding, visual question answering, temporal reasoning, and instruction following, together with embodied cognition tasks including object understanding, spatial reasoning, counting, and egocentric task understanding. Spatially grounded supervision further covers object, area, and affordance localization, trajectory prediction, and manipulation planning, connecting language instructions with explicit locations and interaction targets. We combine this foundational supervision with our visual planning and embodied perception and reasoning data to develop interleaved video–language reasoning while retaining a broad range of understanding capabilities. This combination is intended to support both direct responses to embodied questions and the interpretation of visual evidence produced during task-oriented video prediction.
Frequently Asked Questions
What does the reasoner predict for visual planning? It predicts the next executable subtask, a detailed description of anticipated visual evolution, and an estimated duration.
How is scene memory represented? It uses four independently encoded reference frames, along with a continuous video prefix from the preceding chunk.
What are the three supported generation modes? They are image-to-video, first-and-last-frame-to-video, and cross-chunk video continuation.
Which data contribute to general understanding? The data include a 10% subset of the RynnBrain pretraining corpus, covering general understanding, embodied cognition, and spatially grounded supervision.