Policy steering — guiding the action-sampling process to direct policy outputs toward some desired end — offers a way to use a generalist's reasonable action prior in novel tasks by upweighting relevant behaviors. Steering could allow generalists to make use of knowledge from semantic reasoners, such as humans and large vision-language models (VLMs). For instance, when a policy is learning the novel task of cleaning a kitchen countertop, the knowledge that "sponges are used for wiping spills" could steer the policy to reach for the sponge instead of attempting other behaviors the robot could reasonably do in the scene.
We propose Flow Reversal Steering (FRS): a novel approach that maps coarse reference actions to their noises by passing them through flow policies in reverse. When denoised, this yields actions that are fine-grained and in-distribution for the generalist while staying roughly consistent with the reference action. Given even a rough sketch of robot behaviors (e.g., the general direction needed to reach for a target object), FRS can project that behavior into the generalist's prior to produce a similar fine-grained action.
This mechanism is especially useful for semantic reasoners, such as VLMs, that can roughly infer appropriate robot behaviors but cannot ground them into dexterous low-level actions. FRS moves the onus of emitting robot actions to the generalist, while reasoners can focus on broad, high-level steering. As FRS also gives corresponding noise vectors, it meshes well with latent-noise policies, which steer flow generalists by changing their distribution of input noises. This can enable fast and efficient adaptation via noise-space behavioral cloning (BC) and bootstrap noise-space reinforcement learning (RL) for tasks where exploring via the generalist policy is intractable.
Experimental Setup
We evaluated Flow Reversal Steering by answering three questions: Can FRS improve performance without any training by having VLMs guide generalists toward semantically-reasonable behaviors? Can we use the improved trajectories from FRS to rapidly learn new tasks? Can FRS help generalists more efficiently improve from experience?
FRS Boosts Zero-Shot Performance on Challenging Manipulation Tasks
We first tested if using FRS to refine VLMs' semantic guidance can boost zero-shot performance. We found that directly executing VLM actions is ineffective. This supports the intuition that VLMs struggle with outputting precise low-level actions zero-shot and shows how FRS is not simply reconstructing the VLM actions, but using them to steer toward better — yet still semantically-similar — fine-grained actions from the VLA. Partial noising and sample-and-rank were less performant than FRS, boosting only 4 and 3 hard tasks, respectively. These baselines tend to work well when the VLA already has high probability on good behaviors, not on hard tasks where success is rare, while FRS is able to still learn in this case.
FRS Enables Diffusion Steering via Behavior Cloning
We showed how good trajectories from FRS yield expert noise actions, which can be distilled via Diffusion Steering via Behavior Cloning (DSBC). We focused on online DSBC and presented offline DSBC LIBERO results in the appendix.
Alongside the base policy and zero-shot VLM FRS as baselines, we also compared against running standard BC on the FRS successful trajectories (using the same small architecture as the DSBC noise policy). FRS's successful rollouts can be distilled back into the full VLA as well, but doing so uses much more compute than training an auxiliary policy.
DSBC is sample-, compute-, and time-efficient. In LIBERO, it trains on only 18 rollouts per task on average. On real robots, it needs just 10 rollouts per task to achieve high performance. The policy is likewise small — in total, training takes around 1 GB of GPU memory (as the VLA does not need to be loaded during training), whereas fine-tuning a full VLA requires hundreds of GBs. Due to the model and data size, DSBC policies take under a minute to train.
FRS Accelerates and Improves Reinforcement Learning
We aimed to show how FRS can be used with RL to learn from experience. This allows it to surpass both the fixed performance of zero-shot FRS and distilling FRS's data with DSBC.
Discussion
We introduced Flow Reversal Steering (FRS), a way to convert coarse semantic guidance into precise actions by reversing flow generalist policies. Through extensive simulated and real-world tasks with state-of-the-art VLAs, we showed how FRS allows reasoners, like humans and VLMs, to guide policies toward reasonable behaviors for novel tasks. This enables rapid policy learning through our novel Diffusion Steering via Behavior Cloning (DSBC) method or by bootstrapping DSRL. We hope that FRS provides an alternative paradigm for efficiently improving generalist policies, where learning is not only guided by optimizing reward functions, but task-relevant semantic knowledge as well.
Noise Space Insights
When running flow reversal, more likely noises (i.e., ones closer to 0) tend to map to more in-distribution actions. Flow reversal is better able to find noises that reconstruct to more in-distribution actions (lower reconstruction MSE). Increasing the number of integration steps for flow reversal leads to more accurate reconstruction (due to lower integration error), but at the cost of going more out-of-distribution in noise space.
FRS finds good regions of noise space, and noises near good noises also map to good actions. Noises corresponding to padding elements in action chunks do not seem to affect denoising much. When applying flow reversal, action chunks tend to map to temporally correlated noise chunks, with lower variance across the chunk axis than if sampled from a standard normal distribution.
Takeaways for Running FRS
The key FRS design details motivated by this analysis are: Running FRS by averaging and repeating the noise across the chunk axis does not affect performance. All noise policies are trained to predict such chunk-averaged noises, as it makes learning easier. Averaging and repeating the good reference actions themselves can still find good noises, motivating the use of directional actions for steering. Reference actions should have their padding elements set correctly. Once the underlying noise is computed with flow reversal, the padding noises can be set to zero without affecting performance.
When running flow reversal, there is an inverse correlation between reconstruction fidelity and the resulting noises' magnitudes. Increasing the number of integration steps increases fidelity but makes the noises higher in magnitude (i.e., more out-of-distribution). We err on the side of using fewer steps by choosing the default of 10, as this makes policy inference faster, we want smaller noises for easier noise policy learning, and extremely high action reconstruction fidelity is not always beneficial, especially when reference actions are coarse.
Human Steering Interface
For human FRS experiments, an operator steered the arm via keystrokes: six keys (W/A/S/D/Q/E) for the three translational Cartesian axes (forward/back, left/right, up/down), a key to change gripper state, and a key to defer to the policy. Rotations were never commanded from the keyboard.
The gripper key cycles through three states: open, closed, or noise. In open/closed mode the gripper dimension is driven to the commanded value and held across steering. In noise mode the gripper is left to the policy: the gripper element of the chunk is held at its current value and its corresponding noise element is randomized, so the denoised action is free to open or close based on the observation.
After passing the small reference through inverse kinematics to get joint actions, we tile it until reaching the policy chunk length (15, e.g., repeat the length-5 joint angle chunks thrice). This is what gets passed through flow reversal. In turn, we set the tiled parts of the chunk to zero noise, allowing it to be denoised by the policy based on observation. This is akin to diffusion in-painting, albeit in noise space — we want the steered action to match part of the reference chunk, while the rest of it is "filled in" by what the policy thinks is reasonable.
Frequently Asked Questions
What makes Flow Reversal Steering different from other steering methods? FRS maps coarse reference actions to their noise vectors by reversing flow policies, then denoises them to produce fine-grained actions consistent with the generalist's prior while staying roughly aligned with the reference.
How many rollouts does DSBC need to train effectively? In LIBERO experiments, DSBC trains on only 18 rollouts per task on average, and on real robots it needs just 10 rollouts per task to achieve high performance.
Can FRS work with human operators as the semantic reasoner? Yes, human operators can steer the robot via simple keystrokes for directional commands, with FRS converting those coarse references into fine-grained actions suitable for the generalist policy.
What is the computational cost of training DSBC policies? Training takes around 1 GB of GPU memory (since the VLA does not need to be loaded during training), whereas fine-tuning a full VLA requires hundreds of GBs, and DSBC policies take under a minute to train.