LiMA is an asynchronous multi-scale framework for jointly generating long-term foresight and robotic actions. It employs a dual-system architecture that coordinates generation across distinct temporal scales, enabling rapid physical execution without sacrificing long-horizon planning. To maintain coherence between the two systems, LiMA introduces a Latent Schrödinger Bridge Coupling mechanism that models the transition from abstract intent to motor commands as a continuous probabilistic transport process, tightening the feature coupling between high-level imagination and low-level refinement. This integration is further enhanced by Spatiotemporal Adaptive Modulation, which grounds long-term foresight in real-time physical observations.
LiMA decouples long-range imagination from reactive execution, overcoming generative inference bottlenecks. Its Latent Schrödinger Bridge mechanism with adaptive modulation ensures tight feature alignment between semantic intent and physical action. Evaluation demonstrates competitive performance on real-world tasks while offering computational efficiency for real-time applications.
Preliminaries
The global objective of the generative policy is to model the conditional joint probability distribution to predict a synchronized action chunk over a future temporal horizon. This action chunk encapsulates the coordinated trajectories of both robotic arms and all dexterous fingers.
The omitted formula defines the conditional joint distribution used to predict synchronized future arm and finger actions.
Experiment Setup
Robot Setup
The real-world evaluation platform consists of a bimanual setup featuring two 6-DoF UR5 robotic arms equipped with two 22-DoF SharpaWave five-fingered dexterous hands.
The multimodal perception suite comprises three Intel RealSense D435 cameras: one head-mounted camera capturing the global egocentric view and two wrist-mounted cameras capturing localized coordination views. VIVE Trackers capture the relative spatial trajectories of the human wrists for real-time end-effector control of the UR5 arms. Concurrently, human finger configurations are recorded via MetaGlove Pro gloves and mapped onto the 22-DoF SharpaWave hands through a joint-space kinematic retargeting pipeline.
Baselines and Evaluation Protocol
LiMA is evaluated against four state-of-the-art baselines:
- Gr00T N1.6 and InternVLA-A1, two Vision-Language-Action models.
- VPP and Cosmos-Policy, two World-Action Models.
Success Rate (SR) measures overall task completion, while Progress Success Rate (PSR) provides a fine-grained measure of subtask advancement.
Conclusion
LiMA is an asynchronous diffusion-based framework for bridging long-term imagination and high-frequency reactive execution. It coordinates the Future Dreamer and the Motion Refiner through a dual-system architecture, enabling long-horizon foresight while preserving real-time responsiveness. A Latent Schrödinger Bridge Coupling mechanism further aligns strategic intent with clean execution latents. Extensive real-world experiments show that LiMA improves complex long-horizon manipulation while maintaining robustness and efficient closed-loop control.
This work was supported by the National Natural Science Foundation of China and the Beijing Natural Science Foundation. The authors thank Xiansheng Chen for support with the hardware setup, and Yuqi Cao, Liangwei Hu, and Liwen Tong for their valuable insights and in-depth discussions on the mathematical derivations.
Additional Implementation Details
Latent Token Layout
Following the latent tensor convention of Cosmos-Policy, LiMA represents multimodal information in a five-dimensional latent tensor. Each latent tensor is organized as follows:
The omitted formula specifies the organization of multimodal information within the five-dimensional latent tensor.
Robot System Setup
The real-world robot system includes the system operation pipeline, demonstration collection, and policy execution protocol. The experimental platform consists of two UR5 robotic arms, two SharpaWave five-fingered dexterous hands, three Intel RealSense D435 cameras, VIVE Trackers, and MetaGlove Pro gloves.
Perception System
The three RealSense D435 cameras provide one global view and two local wrist views. The head-mounted camera captures the overall task scene, including target objects, the manipulation workspace, and task progress. The two wrist-mounted cameras are installed near the end-effectors of the left and right robotic arms to provide local visual observations around the contact regions.
All cameras are fixed before the experiments to ensure consistent multi-view observations within the same robot workspace.
Demonstration Collection
During human demonstration collection, the operator uses VIVE Trackers to record the relative spatial trajectories of both wrists. These trajectories are transformed and mapped to end-effector reference trajectories for the two UR5 arms.
MetaGlove Pro gloves record the finger joint configurations of both hands. These configurations are mapped to the 22-DoF SharpaWave dexterous hands through a joint-space kinematic retargeting pipeline.
This process produces synchronized dual-arm trajectories, dual-hand joint trajectories, multi-view visual observations, and language task instructions.
Policy Execution
During policy execution, each policy generates a short-horizon action chunk based on the current observations. The predicted action chunk contains motion commands for the two UR5 arms and joint commands for the left and right SharpaWave dexterous hands.
Before each real-world trial, the experimental scene is reset to a predefined initial state, including the robot initial posture and the corresponding task instruction. All methods are evaluated under the same robot platform, camera configuration, task initialization protocol, and success criteria. This ensures that comparisons between LiMA and the baselines mainly reflect differences in policy design rather than hardware conditions or evaluation settings.
Baseline Settings
LiMA is compared with representative baselines covering integrated world-action modeling, vision-language-action modeling, and video-prediction-based policy learning. All baselines are evaluated under the same real-world robot platform, camera configuration, task initialization protocol, and success criteria as LiMA.
For each baseline, the training or fine-tuning protocol recommended by its original paper is followed whenever applicable. Action outputs are adapted to the same bimanual dexterous robot execution interface.
Cosmos-Policy
Cosmos-Policy is used as an integrated world-action modeling baseline. It maps robot actions into the visual latent space and jointly denoises action latents and image latents within a unified DiT-based denoising process.
Cosmos-Policy is the closest integrated generative baseline to LiMA because both methods model future visual information and robot actions within a shared generative framework.
GR00T N1.6
GR00T N1.6 is used as a representative state-of-the-art vision-language-action baseline. It takes robot visual observations and language instructions as inputs to a vision-language model for semantic understanding. An action head conditioned on the robot state generates continuous actions through a denoising process.
In this setting, the officially recommended 2,000 training steps did not fully expose its best performance on the bimanual dexterous manipulation tasks. Results are therefore reported after training GR00T N1.6 for 50,000 steps on one NVIDIA H100 GPU.
VPP
VPP is used as an inverse-dynamics-model baseline. It uses future visual predictions produced by a video model as conditioning information and feeds them into an action DiT to generate robot actions.
This baseline represents the paradigm of first predicting future visual states and then deriving actions from the predicted visual condition.
InternVLA-A1
InternVLA-A1 adopts a Mixture-of-Transformers architecture that unifies scene understanding, generative foresight, and action modeling within a single framework. The model acquires a rich representation of the environment and uses predicted future visual observations to guide action generation.
To accommodate the full joint configuration of the dual-arm, dual-hand system, comprising two 6-DoF robot arms and two 22-DoF dexterous hands for a total of 56 DoF, the action dimensionality is extended accordingly. The model is initialized from the pretrained InternVLA-A1-3B checkpoint.
Fair Comparison Protocol
All baselines are trained or fine-tuned on the same real-world demonstration dataset whenever task-specific training is required. They use the same task instructions, multi-view robot observations, robot state inputs, and evaluation protocol as LiMA.
Because different baselines adopt different native action representations, their output interfaces are adapted to the same bimanual execution space, including arm motion commands and dexterous hand joint commands.
Evaluation Metrics
The real-world robot experiments primarily use Success Rate, Progress Success Rate, and inference time.
Success Rate
Success Rate measures whether a policy can complete the entire task. For each real-world trial, the trial is considered successful only if the robot completes all predefined task goals in the correct order. SR therefore reflects the final task-level success of each policy.
Inference Time
Inference time evaluates real-time deployability in closed-loop robot control. It denotes the average time required for a policy to generate one executable action chunk from the current observation.
For synchronous inference models, inference time is measured by the forward-pass time of each complete policy inference. For LiMA, the Future Dreamer is updated at a lower frequency while the Motion Refiner runs at the high-frequency control loop. The effective average inference time under asynchronous execution is therefore reported.
Specifically, it is computed as the sum of one Future Dreamer inference time and all Motion Refiner inference times within one asynchronous update cycle, divided by the number of action chunks generated in that cycle.
All inference-time measurements are collected under the same robot platform and execution protocol to ensure a fair comparison.
Generalization Evaluation
For unseen-background and unseen-lighting settings, LiMA shows larger success-rate fluctuations than in the base setting because these perturbations directly affect the visual appearance of the task region.
Nevertheless, LiMA still outperforms the baselines under these conditions. These results suggest that the asynchronous Dreamer–Refiner design and latent intent-to-execution coupling help LiMA maintain stronger robustness when both task-relevant and task-irrelevant visual factors are perturbed.
Frequently Asked Questions
What is LiMA designed to do? LiMA jointly generates long-term foresight and robotic actions while supporting high-frequency reactive execution for dexterous manipulation.
What robot platform is used for evaluation? The platform uses two 6-DoF UR5 arms, two 22-DoF SharpaWave five-fingered hands, three Intel RealSense D435 cameras, VIVE Trackers, and MetaGlove Pro gloves.
Which metrics are used to evaluate the policies? The experiments use Success Rate, Progress Success Rate, and inference time.
How does LiMA support real-time control? Its Future Dreamer operates at a lower frequency while the Motion Refiner runs in a high-frequency control loop through asynchronous execution.