We cast controllable behavior generation as variational inference over a Gaussian behavior latent with a closed-form conjugate posterior and provide a tilt-vs-regression explanatory framework that predicts which channels are identifiable from the return-shrinkage factor. We introduce context-residual return labeling, soft eligibility gates, and contrastive conditioning as extensions to the reward and training pipeline, producing safety controllability substantially above the hard-eligibility ablation while retaining near-ground-truth speed. We present a systematic accounting of the limits in our framework through the scope of drift-paired channel-steering matrices (CSMs) across three return measures, showing that map controllability is coordinate-specific and that CSM values must be interpreted jointly with physical-plausibility guardrails to avoid reward-hacking confounds.
Inference and Controllable Generation
The framework operates with guidance scale a hyperparameter controlling the strength of conditioning. Three regimes are defined: population-marginal (no conditioning), user-controlled (steering active), and prior-sampled (latents drawn from prior without conditioning). A receding-horizon patch scheme enables consistent long-horizon generation, and the full sampler procedure is detailed in the supplementary material.
Experiments
We evaluate the method on the Waymo Open Motion Dataset to answer four questions:
- Does the unconditional path generate realistic behavior?
- Does the learned latent steer per-channel returns with physical plausibility?
- What do ablations without soft eligibility reveal about safety controllability erosion and reward-hacking confounds?
- How sensitive is map controllability to the choice of return measure?
Eligibility Gating and Reward Hacking
Analysis reveals that the early-stage ablation's speed channel-steering matrix value is inflated by reward hacking: it maximizes the speed return by suppressing motion most steered agents stall rather than by maintaining realistic velocities. Channel-steering matrix values must therefore be read jointly with the physical-plausibility guardrails, under which the speed response is physically valid.
Conclusion
Future work includes richer opportunity-aware reward decompositions that separate spatial from temporal map compliance, closed-loop training with contrastive objectives, and stronger latent-conditioning mechanisms beyond one-hot steering.
Acknowledgments
The authors appreciate the Google TPU Research Cloud (TRC) for supporting access to TPUs.
Method Details
Completing the Square
The posterior mean and covariance are obtained through a standard completing-the-square derivation, yielding a closed-form conjugate Gaussian posterior.
Context-Residual Return Labeling
The raw per-channel return conflates agent behavior with scenario difficulty for example, a highway scene yields low offroad penalties regardless of driving style. The context-residual return is defined as the per-channel discounted return residualized against the scene context.
Lane-Centerline Map Reward
As an alternative to the composite offroad-plus-road-edge penalty used in the context-residual return, a lane-centerline return is defined based on the signed lateral offset to the nearest lane centerline.
Soft Eligibility Gates: Decay Formulas
Under soft eligibility, all valid agents receive labels but agents far from hazards contribute negligibly to the discounted return because the decay drives the per-step reward toward zero.
Frequently Asked Questions
How does the method generate controllable behaviors without retraining? It learns a latent variable that conditions the policy during inference using a closed-form conjugate posterior, allowing users to steer specific behavior channels like speed or safety by adjusting the latent.
What is reward hacking and how does the framework prevent it? Reward hacking occurs when a model maximizes a return metric through unrealistic behavior (e.g., stalling to increase speed scores). The framework prevents this by pairing channel-steering matrix values with physical-plausibility guardrails.
Why are context-residual returns needed instead of raw returns? Raw returns conflate agent behavior with scenario difficulty, making it hard to isolate how an agent's actions affect its score. Context-residual returns remove scene-level confounding factors.
What are soft eligibility gates and why are they important? Soft eligibility gates assign decaying weight to agents based on their distance from hazards, rather than a hard cutoff. This preserves safety controllability while avoiding reward hacking that occurs with hard eligibility thresholds.