Introduction
Large datasets and pretrained representations have driven advances in robot imitation learning. Diffusion, flow-matching, and vision-language-action policies fit neural action predictors to expert demonstrations, often using pretrained visual or vision-language backbones. Their success on manipulation and other robot tasks has made neural behavior cloning a powerful approach to learning skills from examples.
This approach nevertheless complicates rapid deployment and behavior revision, and restricts access to robot intelligence. Fitting or adapting a policy can require hours of GPU computation, making frequent updates costly and often requiring server-based compute. Once demonstrations are compressed into network weights, individual actions are difficult to attribute to their supporting examples, and deleting an undesirable demonstration does not remove its influence without updating the fitted model. Generalization to new settings also remains task-dependent. Lookup-based diagnostics show that direct trajectory retrieval can be competitive with diffusion policies, motivating a complementary approach that retains demonstrations at deployment. Our goal is to make robot capabilities easier to deploy and revise when compute and fitting time are limited.
Retrieval policies keep demonstrations explicit, select relevant behavior at inference time, and adapt it to the live query. This exposes the supporting data and makes the demonstration bank a direct point of intervention. The central challenge is query–demonstration mismatch: the robot’s evolving observation–action history need not coincide exactly with a recorded trajectory. Selecting or averaging nearby actions does not explicitly reconcile that history with the behavior that follows it. We instead use the retrieved trajectories as a local model for action continuation, reconstructing the observed past and carrying the resulting coefficients into the demonstrated future.
We introduce Behavior Predictive Control (BPC), which combines a fitted retrieval metric, a regularized Hankel-based action-continuation prior, and a closed-form one-step residual correction. The metric selects windows whose histories are informative about future actions. BPC then reconstructs the live history over those windows and uses the same coefficients to predict their action continuation; a residual fitted on the same support corrects the action to account for nonlinear or higher-order effects. Frozen pretrained visual features extend the representation to visuomotor control. In this work, we find that locality and regularization can address the nonlinear regime, and provide analysis relating next-action error to history reconstruction and local approximation errors.
BPC constructs policies without end-to-end policy training while retaining their supporting trajectories. The retrieved window positions and coefficients also provide an intrinsic estimate of task progress. Simulation and hardware results show competitive performance, generalizing across tasks and embodiments. Reported configurations reduce policy fitting from hours to seconds on consumer GPUs and work even on onboard robot hardware, supporting control at 75 Hz on a Jetson Orin Nano. This combination makes demonstration-based policy construction practical with modest compute while preserving access to the data that determines behavior.
Regularized Local Action Continuation
We adapt the trajectory-completion problem of De Persis and Tesi to behavior cloning: demonstration data constitutes a library of behaviors. Rather than predicting future observations based on a history of observations and executed actions, we seek to predict the expert action that should follow the observed history.
Time indices use parenthesized superscripts; library-column and demonstration indices use subscripts. Consider a closed-loop plant operated by an unknown expert.
Composing Retrieval Distributions
Selection then proceeds as before on this history-aware re-ranked posterior.
Computation Time
Fitting and policy-execution costs of BPC are reported across different dataset scales. Even at massive dataset scales, BPC only takes 3 minutes to fit while remaining well within the memory limits of consumer GPUs. It deploys at 100 Hz on an RTX 4090.
We push BPC to the limits by fitting and deploying on an 8 GB Jetson Orin Nano, which takes less than 3 minutes to fit and was executed onboard the drone at 75 Hz.
Limitations
BPC depends on demonstration coverage and on observation histories that contain enough information to distinguish the required actions. Retrieval cannot resolve ambiguities caused by missing task information, and behavior far from the demonstrated states may require recovery actions absent from the bank. Affine continuation permits extrapolation, but large signed coefficients can amplify approximation error.
Our analysis assumes a locally smooth expert policy and bounds the uncorrected action prior; it does not guarantee closed-loop stability, safety, or task completion after residual correction. Specifically, we find that performance varies across tasks, with weak results on Nut, Stack3, and Pen. Further investigation is required to find mechanisms or dataset-specific quirks that BPC can exploit.
Avoiding end-to-end policy training does not eliminate fitting or data preparation. BPC still fits retrieval and residual components, and retaining the demonstration bank also introduces storage and retrieval costs that grow with the dataset. Although BPC is already compelling for large single-task datasets, future work will be dedicated to massively scaling this paradigm to multitask settings and many more demonstrations.
Acknowledgments
Codex was used in drafting portions of the mathematics in Sections 3 and 4, and in preparing LaTeX formatting. Its image-generation system was used to generate the conceptual robot-arm and drone illustrations.
Observation Encoders
For each task, this encoder is trained from scratch on the selected demonstration subset using an auxiliary behavior-cloning objective for 2,000 updates, batch size 32, and learning rate [learning-rate value not provided in the source text]. The auxiliary model uses two observation steps and predicts 16 action steps; these are encoder-training settings, distinct from BPC’s 10-step history and continuation.
Only the point-cloud encoder is retained for BPC, and it is frozen during retrieval and residual fitting and deployment. Thus, the reported 3D pipeline includes task-specific encoder training; it does not use released SDP weights or the public PointNet++ alternative. Frozen pretrained PointNet latents achieve slightly lower success rates on these dexterous tasks.
Retrieval and Residual Hyperparameters
Two distinct random-feature maps are used: the retrieval map in Linear Discriminant Analysis and the residual-correction map. Both contain 16,384 features; the reduced retrieval representation has rank 70. The retrieval fit uses up to 8,192 anchors. These dimensions should not be confused with the image or point-cloud embedding dimensions.
The archived retrieval configuration uses the entropy-teacher Fisher-LDA fit. The affine readout uses the exact solver with a sum-to-one constraint and no additional [regularization-term value not provided in the source text] penalty.
Frequently Asked Questions
What is Behavior Predictive Control? Behavior Predictive Control is a retrieval-based policy construction method that uses demonstrated trajectories for action continuation and applies a residual correction.
Does BPC require end-to-end policy training? No. BPC avoids end-to-end policy training while fitting retrieval and residual components and retaining the demonstration bank.
How fast can BPC run on robot hardware? BPC supports control at 75 Hz on a Jetson Orin Nano and deployment at 100 Hz on an RTX 4090.
What are the main limitations of BPC? BPC depends on demonstration coverage and informative observation histories, and it does not guarantee closed-loop stability, safety, or task completion after residual correction.