A predictive model of how a scene will evolve is useful for robot planning, representation learning, and data generation. Video prediction offers a general interface because it retains the geometry and appearance needed by downstream visual policies. However, real robot videos present an uncomfortable combination: most pixels are static, small articulated parts determine the action, and the same natural-language instruction can admit visually distinct trajectories. A model can therefore achieve competitive global image metrics by copying the last frame, without learning the motion that matters.
Introduction
AcrossVAM1.0 assigns different responsibilities to different representations. Semantic particles describe the geometry of the whole robot, arm, and gripper; a compact Transformer predicts their future states; and a separate causal appearance stream restores detail. Language is injected only into the dynamics through feature-wise modulation, which makes its effect measurable at the trajectory level. The dense branch observes the last context frame but never a future frame, so future motion cannot leak through the appearance path.
We introduce a semantically anchored particle world model that reduces real-robot video dynamics to a compact trajectory problem. Its dynamics core has only 0.28M trainable parameters and exposes part-level motion for diagnosis.
We propose a causal motion–appearance decoder that combines predicted particle geometry with context-only dense appearance and explicitly controls the trade-off between sharp static regions and moving-object fidelity.
We evaluate against persistence, shuffled-language and random controls, oracle particles, multiple seeds, and per-robot splits. The evaluation identifies both a reliable motion benefit and the still-limited degree of language grounding.
Problem Formulation
An object-centric state represents the scene in terms of its relevant objects and parts.
The dynamics function predicts how the object-centric state changes over time.
The decoder renders the predicted motion together with context appearance.
Frozen Semantic Particle Codec
For each frame, a frozen SAM3-DLP encoder receives RGB and VRS part masks and extracts semantic particles corresponding to the whole robot, arm, and gripper, plus a spatial background latent.
Each semantic particle represents the encoded state of one robot part or scene component.
Unordered proposals require temporal correspondence. For adjacent frames, we form correspondences and obtain matched particle identities using Hungarian matching. The bar denotes scale-normalized coordinates. In our offline VRS cache, fixed semantic part identifiers already determine the permutation; we bypass matching there and retain it for proposal-based inference.
Text-Assisted Particle Dynamics
The dynamics input concatenates geometry, first-order velocities, depth, presence, and learned slot and time embeddings. A causal spatio-temporal Transformer alternates temporal reasoning within each slot with interactions among robot parts. It predicts residual updates autoregressively.
The residual update specifies the predicted change to each particle state.
The text feature is produced by a frozen OpenCLIP ConvNeXt-L/320 text tower.
At each Transformer block, trainable FiLM maps inject instruction semantics.
Feature-wise linear modulation scales and shifts intermediate dynamics features using language-conditioned parameters.
FiLM projections are initialized to zero, so optimization starts from unconditioned dynamics rather than a randomly perturbed model. We combine one-step transition and autoregressive rollout supervision.
The training objective combines next-step prediction with supervision over multi-step predicted trajectories.
Representation and Dynamics
Adding a residual code raises reconstruction PSNR by 7.01 dB, motivating a dual stream rather than forcing texture through object state. The small dynamics model reliably beats trajectory persistence across seeds. Diagnostics show that residualized box coordinates and raw appearance features introduce shortcuts; removing them and using stride-three motion raises error reduction from low single digits to 21.0%.
Does the Model Use Language?
The positive multi-seed gap, together with near-zero gaps for kinematic pseudo-text and random labels, suggests that FiLM captures a real semantic signal. The effect is nevertheless small and below the 10% target. We therefore describe AcrossVAM1.0 as text-assisted, not instruction-controlled. The lower verb recall of automatically generated captions also explains why a larger language encoder does not automatically produce a larger causal effect.
The language-conditioned particle model improves on Franka (+21.1%, 136 windows), WidowX (+19.0%, 68), and UR5 (+18.4%, 8), but degrades on Mobile ALOHA (60 windows), Google Robot (32 windows), and several small subsets. The non-Franka aggregate remains positive (+10.0%, 232 windows), but strict leave-one-robot-out Mobile ALOHA experiments do not beat persistence.
Conclusion
We introduced AcrossVAM1.0, a compact text-assisted world model that treats robot video prediction as semantic particle dynamics plus causal appearance synthesis. The model improves trajectories and motion-region prediction while residual delivery recovers much of the static visual fidelity. Strong controls show both what works—low-dimensional motion modeling—and what remains unresolved—perceptual delivery, language dependence, and cross-robot transfer. This separation provides a foundation for interpretable video action models without hiding failure modes behind aggregate image metrics.
Frequently Asked Questions
What is AcrossVAM1.0? AcrossVAM1.0 is a compact text-assisted world model for robot video prediction based on semantic particle dynamics and causal appearance synthesis.
What do the semantic particles represent? They represent the geometry of the whole robot, arm, and gripper, along with a spatial background latent.
How is language incorporated into the model? Language is injected into the particle dynamics through feature-wise linear modulation using features from a frozen OpenCLIP text tower.
Does the model fully control robot motion from instructions? No. The measured language effect is positive but small, so the model is described as text-assisted rather than instruction-controlled.