Closed-loop policy evaluation is subject to closed-loop drift. A scenario scripted before execution with predetermined actor routes, speeds, or behaviors may not preserve its intended interaction once the ego policy responds in closed loop. Reactive actors are not enough, nor are scenario triggers. A scenario orchestrator addresses this gap by continuously monitoring the joint traffic state and jointly adapting non-ego actors through re-timing, re-routing, and role reassignment. It solves a constrained optimization at each step so that the intended interaction remains attainable while interventions stay limited and traffic remains feasible.
Scripted Execution
Scenario realization is measured by whether a rollout satisfies a sequence of scenes. This is a common paradigm in scenario-based testing, where a scenario is represented as the temporal development of a sequence of scenes starting from an initial scene. Scene graphs encode the structure and relationships between entities in a scene. Recent testing work uses them to generate tests from initial scenes and continuations allowed by a specification.
Data-Driven Generation
Score-based metrics measure how safety-critical the actors are to the ego, but cannot reflect whether a rollout realizes a specific scenario.
Method
A scalable multi-agent traffic simulation controller addresses the problem of scenario orchestration.
Constrained Optimization
The approach solves a constrained optimization problem in closed loop with receding-horizon control.
Scenario Representation
A scenario is modeled as a list of actors, each with a sequence of maneuvers. A maneuver is a discrete action from the set left_turn, go_straight, right_turn, lane_change_left, and lane_change_right, with continuous parameters such as speed and turn radius. Execution works in parallel across actors and, for a given actor, serially in the order of the maneuver sequence.
Experiments
The experiments address two research questions:
- RQ1: Does the method realize scenarios more effectively than other methods? This examines how well-suited different methods are to targeted realization of a predetermined test intent.
- RQ2: Does the method stress-test policies more effectively than other methods? This examines how well different methods can identify and induce failures in ego policies conditioned on intent.
Conclusion
Closed-loop policy evaluation is subject to closed-loop drift. A scenario scripted before execution with predetermined actor routes, speeds, or behaviors may not preserve its intended interaction once the ego policy responds in closed loop. Reactive actors are not enough, nor are scenario triggers. The scenario orchestrator continuously monitors the joint traffic state and jointly adapts non-ego actors through re-timing, re-routing, and role reassignment. At each step, it solves a constrained optimization so that the intended interaction remains attainable while interventions stay limited and traffic remains feasible.
Experiments in CARLA, across scenario families and ego policies, show that this orchestration realizes targeted interactions more reliably than scripted and learned actor-level baselines, including under changes in initial conditions and ego behavior. By scoring scenario realization separately from the ego’s response, the evaluation can first ask whether the intended interaction occurred and only then how the policy behaved under that interaction.
The current design is limited by its reliance on manually specified traffic behaviors, such as ranking actors by their likelihood of carrying the intended action. This approach will not scale as scenarios become more complex and open-ended. Integrating data-driven actor models with the objectives and constraints is left to future work.
Acknowledgments
Generative AI tools, including AI agents, were used for engineering and implementation of the codebase.
Frequently Asked Questions
What is scenario orchestration? It is a method that monitors the joint traffic state and adapts non-ego actors to keep an intended interaction attainable.
How does the method adapt traffic actors? It uses re-timing, re-routing, and role reassignment, with constrained optimization solved at each step.
How is scenario realization measured? A rollout is considered realized when it satisfies a sequence of scenes.
What limitation do the authors identify? The design relies on manually specified traffic behaviors, which may not scale to more complex and open-ended scenarios.