Deploying reactive visuomotor policies on physical robots requires real-time inference, but modern architectures incur significant latency overheads from processing high-dimensional visual observations. We propose a reactive real-time flow policy that re-architects the processing pipeline to enable fast, reflex-like responses while maintaining task performance. Our approach decouples visual processing from control, using a lightweight proprioceptive pathway that operates at 1 kHz, with visual context provided asynchronously.
Architecture Overview
The core insight is that reactive behaviors rely primarily on proprioceptive feedback, while visual information provides spatial context at a coarser timescale. We design a dual-stream architecture: a fast proprioceptive policy running at kilohertz frequencies, and a slower visual processing stream that updates at 10 Hz. The proprioceptive stream uses a diffusion policy formulation conditioned on the latest visual features.
Training Methodology
We train the visual encoder separately using behavioral cloning on demonstration data, then freeze it during proprioceptive policy training. The proprioceptive policy is trained using a flow-matching objective that learns to generate smooth, reactive trajectories conditioned on the current proprioceptive state and the most recent visual embedding. This decoupled training approach enables efficient learning without end-to-end backpropagation through the entire system.
Real-Time Implementation
The system runs on a standard desktop with a GPU for visual inference and a separate CPU thread for the proprioceptive policy. Communication between threads uses a lock-free ring buffer to minimize latency. We achieve end-to-end inference times under 0.5 ms for the proprioceptive stream, enabling reactions within a single control cycle of the robot.
Experimental Results
We evaluate on four challenging manipulation tasks: dynamic object catching, obstacle avoidance during reaching, peg insertion with perturbations, and cloth folding with moving obstacles. Our method matches or exceeds the task success rates of prior state-of-the-art approaches while reducing reaction latency by two orders of magnitude.
| Task | Ours (latency) | Prior SOTA (latency) | Ours (success) | Prior SOTA (success) |
|---|---|---|---|---|
| Dynamic catching | 0.3 ms | 42 ms | 91% | 88% |
| Obstacle avoidance | 0.4 ms | 38 ms | 94% | 92% |
| Peg insertion | 0.5 ms | 45 ms | 97% | 96% |
| Cloth folding | 0.4 ms | 40 ms | 85% | 83% |
Ablation Studies
We conduct ablation experiments to understand the contribution of each design choice. Removing the asynchronous visual updates degrades performance by 15-20% on dynamic tasks. Increasing the proprioceptive policy update rate from 100 Hz to 1 kHz improves task success by 8% on catching tasks, confirming the importance of high-frequency control for reactive behaviors.
Limitations
Our approach has several limitations. First, we do not address sources of latency external to the model itself, such as communication delays between the inference server and the robot client. Second, we keep the base policy architecture unchanged; designing it to emphasize proprioceptive features more strongly (e.g., dedicated attention heads for proprioceptive tokens) could further amplify reactivity, and we leave this direction to future work.
Frequently Asked Questions
What makes this approach different from other real-time robot control methods? This work decouples visual processing from control, enabling a lightweight proprioceptive policy running at 1 kHz while visual context updates asynchronously at only 10 Hz.
How does the training process work with separate visual and proprioceptive streams? The visual encoder is trained first via behavioral cloning, then frozen. The proprioceptive policy is trained separately using flow matching, conditioned on the frozen visual embeddings and current robot state.
What hardware is required to achieve the reported latencies? The system runs on a standard desktop PC with a GPU for visual inference and uses a CPU thread for the high-frequency proprioceptive policy, with inter-thread communication via lock-free ring buffers.
Can this approach be applied to robots with different sensing modalities? The dual-stream architecture is general—any high-dimensional sensor that provides context at slower timescales can be processed asynchronously, while a fast low-dimensional feedback loop handles reactive control.