Researchers have developed FM-VLA, a vision-language-action model that adds a dedicated force-based memory module, enabling robots to reason over sequences of physical contacts during complex manipulation tasks. This approach allows a humanoid robot to succeed at tasks like the classic "shell game" — lifting cups to find a hidden block — where visual information alone is ambiguous and the robot must remember which cup it has already touched.
What the Researchers Built
FM-VLA stands for Force-based Memory for Vision-Language-Action models. The core innovation is a force memory module — a variational autoencoder (VAE) trained on sequences of 6-axis wrench signals (3-axis force + 3-axis torque) recorded from wrist-mounted sensors on a bimanual humanoid robot (AgiBot G1). The VAE learns a compact, task-agnostic representation of the robot’s contact history. This latent representation is then injected as "force memory tokens" into a standard VLA policy, allowing the model to remember past physical interactions even when visual observations are limited or identical (e.g., two visually identical cups).
The system was tested on three contact-rich tasks requiring temporal reasoning: the cup-and-block "shell game" (16–24 steps), a peg insertion task, and a drawer opening task. The force memory tokens enable the policy to distinguish between "already checked" and "not yet checked" cups without repeated visual inspection.
Key Results
In quantitative evaluations of 18 trials per task (9 for each block position in the cup task), FM-VLA significantly outperformed three baselines:
- Vision-only baseline (no force memory): succeeded on less than 30% of trials.
- Vision + single-frame force conditioning (no temporal memory): succeeded on ~50% of trials, often failing when visual cues were ambiguous.
- Vision + vision-based memory (frame stacking or recurrent vision): improved over vision-only but still failed at distinguishing physically identical states.
- FM-VLA (full model): achieved over 85% success rate across all tasks, with negligible inference overhead (less than 2 ms added latency per step).
The force memory VAE was trained in a task-agnostic manner on wrench histories from all tasks combined, then frozen and its latent tokens (8 tokens per window) fed into the VLA action expert during fine-tuning. Ablation studies confirmed that removing the force memory or replacing it with proprioception-only signals caused performance to drop to baseline levels.
How It Works
FM-VLA extends a standard vision-language-action model with a two-stage training pipeline.
Stage 1: Force memory VAE pretraining. The VAE is trained on 100 Hz 6-axis wrench time series recorded from both wrist sensors during all demonstration tasks. The loss function is a masked-ELBO (evidence lower bound) with free-bits regularization on each latent dimension. This produces a general-purpose latent space that encodes temporal patterns of contact events — e.g., the characteristic force profile of lifting a cup, setting it down, or grasping a block. The encoder compresses a sliding window of wrench history into 8 latent tokens.
Stage 2: Integration into VLA policy. The frozen VAE encoder is attached to the policy alongside the existing visual encoder (3 RGB camera streams) and proprioception encoder. The 8 force memory tokens are concatenated with action tokens in the action expert transformer. The policy is finetuned end-to-end on task demonstrations, while the VAE encoder remains frozen. This ensures the force memory representation remains stable and does not overfit to task-specific details.
At inference time, the robot receives language instructions (e.g., "Find the block under the cup and pick it up with left hand."), visual observations, and the current wrench stream. The VAE encoder outputs memory tokens that inform the policy about past contacts. For example, after lifting the first cup and finding no block, the wrench signature of "cup set down" is encoded, and the policy knows not to re-lift that cup.
The model was evaluated on an AgiBot G1 bimanual humanoid with two 7-DoF arms, grippers, and 6-axis force/torque sensors at each wrist. Training used 50k finetuning steps with a WSD schedule, batch size, and image dropout.
Why This Matters for Robotics
Force sensing is cheap, robust, and available on many humanoid robots and collaborative robots. The ability to use force history as a form of temporal memory unlocks tasks that require understanding the sequence of physical events — not just the current pose or visual scene. This is critical in manufacturing, assembly, and logistics where robots must inspect, sort, or manipulate parts that look identical but differ in state (e.g., empty vs. filled containers).
FM-VLA's approach is particularly relevant for used industrial robots being retasked for flexible operations, where adding a low-cost force sensor and this memory module could enable new capabilities without expensive vision upgrades. For warehouse robots handling packages, force memory could track which items have been picked, even when barcode scanners fail.
Limitations and Open Questions
The current VAE uses a fixed bottleneck of 8 latent tokens, which may limit performance on very long-horizon tasks with hundreds of contact events. Hierarchical or adaptive compression could be needed for tasks like furniture assembly with dozens of pick-and-place steps.
The VAE is trained solely on the demonstration dataset for the three tasks. Pretraining on large-scale robot datasets with diverse force/torque recordings (e.g., from public manipulation datasets) could improve generalization to new environments or sensor types.
Finally, the force memory is extracted from 6-axis wrench sensors mounted at the wrist. Not all robots have such sensors; adapting FM-VLA to use joint torque sensors or even current sensing would broaden applicability.
Frequently Asked Questions
What is a force memory VAE? It is a variational autoencoder trained to compress sequences of force/torque signals into a small set of latent tokens that represent the robot’s recent physical interaction history.
How does FM-VLA differ from standard VLAs? Standard VLAs process current visual and proprioceptive states; FM-VLA adds a dedicated memory module that encodes temporal force patterns, enabling the robot to reason about past contacts even when visual input is ambiguous.
What hardware is needed to run FM-VLA? The method requires a 6-axis force/torque sensor at the robot wrist (or similar contact sensing) and a vision-language-action capable compute platform. It was demonstrated on an AgiBot G1 bimanual humanoid.
Can FM-VLA work with other robot architectures? Yes, the force memory module is model-agnostic and can be integrated into any VLA policy that accepts additional token sequences, as long as force/torque data is available.
Conclusion
FM-VLA demonstrates that adding a dedicated force-based memory to vision-language-action models dramatically improves performance on contact-rich manipulation tasks requiring temporal reasoning. By leveraging cheap and ubiquitous force/torque sensors, the system achieves robust physical memory with minimal computational overhead, pointing toward more capable and reliable robots for real-world assembly and inspection.