DeepScenario: City Scale Scenario Generation for Automated Driving System Testing & Evaluation

Liu, Henry; Feng, Yiheng · 2023 · ROSA P / University of Michigan. Center for Connected and Automated Transportation

archive: archived pipeline: cataloged verified

Get this paper ↗ (full text — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the critical inefficiency in testing and evaluating the driving intelligence of autonomous vehicles (AVs). Current methods rely on naturalistic driving environments (NDE), which suffer from the "curse of dimensionality" and the extreme rarity of safety-critical events. Consequently, demonstrating AV safety requires hundreds of millions of miles of testing, a process that is computationally prohibitive. The authors propose DeepScenario, a method that generates a Naturalistic and Adversarial Driving Environment (NADE) to accelerate evaluation by multiple orders of magnitude while maintaining statistical unbiasedness. The methodology combines data-driven modeling with reinforcement learning. First, the authors construct a baseline NDE using naturalistic driving data from the University of Michigan’s Safety Pilot Model Deployment and Integrated Vehicle-Based Safety System programs. They model vehicle maneuvers as a Markov Decision Process, sampling actions from empirical distributions derived from real-world data to ensure realistic traffic behavior. To create the NADE, the authors apply importance sampling theory to identify "principal other vehicles" (POVs) whose maneuvers pose the highest risk to the AV. They calculate a "maneuver challenge" metric using surrogate models and reinforcement learning to estimate the probability of a crash resulting from specific background vehicle actions. At critical moments, the NADE twists the maneuver distribution of the POV to increase adversarial interactions, while other vehicles continue to follow naturalistic behaviors. The study validates the approach using high-fidelity simulations in CARLA and a highway traffic simulator. Two distinct AV agents—one based on standard driving behavior models and another trained via deep reinforcement learning—were tested in both NDE and NADE settings. Results demonstrate that NADE generates significantly more safety-critical events, such as cut-ins and lane conflicts, compared to NDE. Crucially, the accident rates and performance metrics observed in NADE remained statistically consistent with those in NDE, confirming that the adversarial adjustments did not introduce bias. The NADE approach accelerated the evaluation process by multiple orders of magnitude, achieving equivalent statistical accuracy with a fraction of the simulation miles required by traditional methods. The significance of this work lies in its ability to solve the rare-event estimation problem in high-dimensional spaces. By isolating critical variables and applying targeted adversarial adjustments, DeepScenario provides a scalable, efficient framework for AV testing. This method allows developers to systematically evaluate driving intelligence and identify corner cases without the prohibitive computational costs of pure naturalistic simulation, thereby facilitating faster and more rigorous safety validation for autonomous systems.

Key finding

The proposed Naturalistic and Adversarial Driving Environment accelerates autonomous vehicle evaluation by multiple orders of magnitude compared to naturalistic driving environments while maintaining unbiased accuracy.

Methodology

simulator

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via bulk_ingest_rosap on 2026-05-23 (6 acquisition events logged).

Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• situational awareness
• naturalistic crash near crash

Information type

What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).

• Methodological Resource: dataset resource, tool software
• Theoretical Contribution: computational model