Determining the Validity of Simulation Models for the Verification of Automated Driving Systems
DOI: 10.1109/access.2023.3316354
archive: archived pipeline: cataloged verified
Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)
Summary
This paper addresses the critical challenge of validating simulation models used for verifying automated driving systems (ADS). As physical testing requires hundreds of millions of kilometers to ensure safety, the industry increasingly relies on virtualization. However, this shift necessitates rigorous methods to ensure simulation environments accurately represent reality. The authors focus on operational validity, specifically determining whether a simulation behaves indistinguishably from the real world regarding relevant aspects of logical scenarios. The work aims to provide a non-parametric, objective method to measure the distance between simulation traces and real-world data, supporting safety argumentations without relying on strong assumptions about data distributions. To achieve this, the authors propose modeling sets of discretized simulation runs as time-homogeneous Markov chains. This approach captures the temporal evolution of physical objects within scenarios. The core methodological contribution is the application of the Maximum Mean Discrepancy (MMD) to compare these Markov chains. Unlike traditional tests that require estimating full distributions or are limited to univariate cases, MMD allows for direct comparison of samples in a non-parametric manner. The authors define a computable distance measure based on MMD to assess the similarity between the distributions of states and transition probabilities of two Markov chains. This method is designed to be objective, generalize deterministic comparisons, and reflect differences across all moments of the distributions, fulfilling most desirable properties for validation metrics. The study evaluates the feasibility of this approach through two experiments using the Method of the Manufactured Universe. In this setup, one simulation acts as the "reality" with known true values, while another serves as the model to be validated. This allows for a controlled assessment of the method's ability to detect discrepancies. The authors demonstrate that by comparing the start distributions and transition matrices of the Markov chains, the MMD-based metric can effectively quantify the representativeness of the simulation. The results indicate that the method can successfully distinguish between simulations that accurately model the target behavior and those that do not, provided that all relevant variables are observable and no crucial hidden factors are omitted. The significance of this work lies in providing a rigorous, statistical framework for simulation validation in the context of ADS safety. By enabling the comparison of simulation traces with real-world counterparts or reference simulations, the method supports the generation of quantitative evidence for safety cases. This addresses a key gap in scenario-based verification, where ensuring the validity of synthetic data is essential for regulatory acceptance. The proposed approach offers a scalable solution for validating complex, high-dimensional simulation environments, thereby facilitating the reliable use of virtual testing in the development and homologation of automated vehicles.
Provenance
The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.
| Stage | Outcome | Tool | Model | Prompt | Attempts | Completed |
|---|---|---|---|---|---|---|
| discover | success | Crossref | — | — | 1 | 2026-06-18 |
| archive | success | unpaywall | — | — | 2 | 2026-06-25 |
| extract | success | cached | — | — | 2 | 2026-06-26 |
| clean | success | clean | — | — | 1 | 2026-06-20 |
| chunk | success | chunk | — | — | 1 | 2026-06-20 |
| embed | success | embed | Qwen/Qwen3-Embedding-8B | — | 1 | 2026-06-20 |
| enrich | success | openalex | — | — | 1 | 2026-06-20 |
| promote | success | — | — | — | 1 | 2026-06-18 |
| summarize | success | llm | qwen3.6-27b-prismaquant | summ-v5 | 1 | 2026-06-26 |
| tag | success | vector_similarity | — | — | 6 | 2026-06-20 |
| verify | success | — | — | — | 1 | 2026-06-26 |
Summary generated by qwen3.6-27b-prismaquant on 2026-06-26; verification: verified.
Topics
Ranked by relevance to this paper. Hover a topic for its definition.
Information type
What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).
- Methodological Resource: validation psychometrics, tool software
- Theoretical Contribution: computational model