We present DynaFLIP, a dynamics-aware 3D flow-language-image pre-training framework that pushes motion understanding upstream into perception. By jointly aligning image transitions, language, and 3D flow through a simplex-based formulation—augmented with a cosine regularizer and a contrastive framework to resolve optimization pitfalls—DynaFLIP learns visual representations that focus on control-relevant regions. Across simulated and real-world manipulation, DynaFLIP transfers strongly as a reusable visual backbone and consistently outperforms baselines, with especially large gains under visual, spatial, and semantic distribution shifts. Our results indicate that robot generalization improves when visual representations are trained to encode not just what is present, but how the world changes under action.
Does DynaFLIP Learn Dynamics-Aware and Control-Relevant Representations?
Experiment setup. We first verify our central claim that DynaFLIP's pre-training yields dynamics-aware representations that preserve control-relevant information. We analyze pre-trained image encoders on MetaWorld and RLBench: each encoder remains frozen, and only a lightweight three-layer MLP policy is trained on top, ensuring that downstream performance reflects representation quality rather than policy capacity.
Together, the quantitative and qualitative results show that DynaFLIP learns dynamics-aware representations that preserve control-relevant information and focus on regions critical for manipulation.
Experimental Evaluation
In this section, we evaluate DynaFLIP through extensive experiments in both simulation and the real world. Through these experiments, we aim to answer the following questions:
- Does DynaFLIP learn dynamics-aware representations that preserve control-relevant information for manipulation?
- Do dynamics-aware representations improve downstream policy learning compared to strong baselines?
- Can DynaFLIP improve real-world manipulation under both in-distribution and out-of-distribution settings?
- Which design choices in DynaFLIP are most critical to its performance?
Related Work
Both paradigms, however, learn primarily from static data and therefore lack dynamics awareness. This limitation matters for manipulation, where success depends on how scenes change under interaction, not only on object and instruction recognition. DynaFLIP addresses this gap by aligning three transition-centric modalities—image transitions, language, and 3D flow. These signals allow the encoder to focus on control-relevant regions rather than visually salient but task-irrelevant areas.
Dataset Generation Pipeline
We follow the unified data generation pipeline of TraceForge with several modifications tailored to our setting. Our pipeline converts raw videos into aligned image-language-3D flow triplets. Compared with the original pipeline, we omit event chunking and speed retargeting, and instead directly sample frames from each video so that the effective temporal resolution is approximately matched across datasets collected at different frame rates while preserving the original motion timing.
Language instruction generation. For each sampled trajectory, we generate language instructions that describe the underlying manipulation intent using a vision-language model (VLM). The VLM takes as input a small set of representative frames sampled from the trajectory together with a prompt asking it to describe the task in three forms: a short imperative instruction, a detailed natural-language description, and a multi-step instruction that decomposes the task into sequential subgoals.
LIBERO Evaluation
We evaluate DynaFLIP on five LIBERO suites: LIBERO-90, LIBERO-Goal, LIBERO-Object, LIBERO-Spatial, and LIBERO-Long. LIBERO-90 contains 90 tasks, while each of the other four suites contains 10 tasks with 50 demonstrations per task.
For visual input, we use only third-person RGB observations and exclude gripper-view images. The image encoder output serves as the visual conditioning vector for the diffusion policy. For CNN-based encoders, we obtain the global image feature by applying global average pooling to the final feature map of the ResNet backbone. For ViT-based encoders, we concatenate the classification token with the average-pooled patch tokens to form the image feature.
Language instructions are encoded using the corresponding text encoder when available. For R3M, VC-1, and DINOv2, which do not provide native text encoders, we use the CLIP text encoder. CLIP, LIV, and DynaFLIP use the classification token representation as the sentence-level language feature, whereas SigLIP uses mean pooling over all token embeddings.
Training and evaluation protocol. Our primary LIBERO setting follows a reusable-encoder protocol: both the image and language encoders remain frozen, and only the diffusion policy is trained. This setting directly evaluates whether each pre-trained representation can transfer to downstream policy learning without task-specific encoder adaptation. As an additional comparison, we also report a LoRA setting that adapts both encoders jointly with the diffusion policy.
For each LIBERO suite, we train a separate diffusion policy using demonstrations from that suite and evaluate it on the corresponding suite. Each method is trained for 200 epochs. Every 20 epochs, we evaluate the policy using 20 rollouts per task. We then select the best-performing checkpoint across training and report its average rollout success rate.
Generalized Simplex Volume
For an n-modal tuple, let v_i be L2-normalized modality embeddings, and define e_i = v_i - v_n as simplex edge vectors. Let G be the Gram matrix of the simplex edge vectors. The generalized simplex volume is defined as Vol(S) = sqrt(det(G)). A smaller value indicates that the modality embeddings form a lower-volume configuration in the shared latent space, reflecting stronger joint alignment across modalities. In the three-modal setting used by DynaFLIP, this quantity reduces to the triangle area defined in Eq. ( 1 ), and we focus on this case in the analyses that follow.
Why Simplex-Volume Alone Is Insufficient
We analyze the simplex-volume alignment in the three-modal case, where the objective reduces to the triangle area defined in Eq. ( 1 ). This setting allows us to characterize the alignment gradient and reveal two key limitations: ambiguity of low-volume configurations and conflicting alignment gradients.
Low simplex volume does not necessarily imply pairwise alignment among all modalities. The simplex volume can vanish even when some modality pairs remain far apart—for example, when a subset of embeddings collapses together, or when all embeddings become nearly collinear.
For example, we illustrate this in the three-modal case. Consider unit vectors in R².
Conclusion
We present DynaFLIP, a dynamics-aware 3D flow-language-image pre-training framework that pushes motion understanding upstream into perception. By jointly aligning image transitions, language, and 3D flow through a simplex-based formulation—augmented with a cosine regularizer and a contrastive framework to resolve optimization pitfalls—DynaFLIP learns visual representations that focus on control-relevant regions. Across simulated and real-world manipulation, DynaFLIP transfers strongly as a reusable visual backbone and consistently outperforms baselines, with especially large gains under visual, spatial, and semantic distribution shifts. Our results indicate that robot generalization improves when visual representations are trained to encode not just what is present, but how the world changes under action.
Frequently Asked Questions
How does DynaFLIP align three different modalities during pre-training? DynaFLIP uses a simplex-based alignment formulation that jointly minimizes the volume formed by image transition, language, and 3D flow embeddings in a shared latent space, augmented with a cosine regularizer and contrastive framework.
What makes DynaFLIP different from existing vision-language pre-training methods for robotics? Unlike methods trained on static data, DynaFLIP explicitly incorporates dynamics awareness by aligning transition-centric modalities (image transitions and 3D flow) with language, focusing on control-relevant regions rather than visually salient but task-irrelevant areas.
How does DynaFLIP perform under distribution shifts in real-world manipulation? DynaFLIP shows especially large gains under visual, spatial, and semantic distribution shifts compared to baselines, demonstrating strong generalization capabilities when deployed in out-of-distribution settings.
What are the failure modes of using simplex-volume alone for modality alignment? Low simplex volume can occur even when some modality pairs remain far apart, such as when a subset of embeddings collapses together or when all embeddings become nearly collinear, leading to ambiguous alignment.