Validating Simulation Environments for Automated Driving Systems Using 3D Object Comparison Metric
DOI: 10.1109/iv51971.2022.9827354
archive: archived pipeline: cataloged verified
Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)
Summary
This paper addresses the critical challenge of validating simulation environments for Automated Driving Systems (ADS). While simulation-based testing is essential for ADS verification and validation due to the impracticality of driving billions of miles in the real world, simulation results are only trustworthy if the virtual environment accurately represents reality. The authors propose a novel methodology to quantify the fidelity of simulation platforms by comparing 3D object data from real-world LiDAR scans against simulated LiDAR sensor outputs. The study employs a three-stage methodology: data collection, virtual map synthesis, and comparison. Real-world data was collected using a 360-degree LiDAR mounted on an autonomous pod driven along a route on the University of Warwick campus. Six different scan configurations were generated by varying vehicle speed (2 m/s and 4 m/s) and LiDAR rotation rates (300, 600, and 1200 rpm). The real-world point clouds were converted into voxel environments, which were then used to generate simulated LiDAR scans. To compare the real and simulated maps, the authors adapted a histogram-based comparison method. This approach calculates pairwise Euclidean distances between sampled points on the object surfaces, normalizes these distances, and stores them in histograms. The dissimilarity between the real and simulated maps is then quantified using the Minkowski distance between these histograms. This method was chosen for its ability to handle large, complex, non-convex objects without requiring manual alignment or prior geometric knowledge. The results demonstrate that the proposed histogram method provides consistent and symmetrical comparison scores, outperforming the traditional Chamfer distance metric, which exhibited significant outliers and asymmetry. The average dissimilarity score across all configurations was 0.247. The analysis revealed that simulation fidelity is influenced by scene complexity; areas containing vehicles and buildings yielded lower dissimilarity scores (higher accuracy) compared to areas with trees, indicating the simulation is currently more viable for rigid structures than organic shapes. Additionally, the study identified that the 4 m/s / 1200 rpm configuration provided the best match between real and simulated data, while scans with higher noise levels (e.g., 600 rpm) showed poorer matching despite having more data points. The significance of this work lies in providing a robust, automated metric for validating ADS simulation environments. By quantifying the similarity between virtual and real-world sensor data, this approach supports the credibility assessment frameworks required for ADS safety assurance. The method is generalizable to other 3D sensors, such as cameras and radar, and can be adapted to evaluate simulation fidelity across different Operational Design Domains (ODDs). This contributes to the broader field by offering a concrete tool to ensure that simulation-based testing results are representative of real-world conditions, thereby strengthening the safety case for automated driving systems.
Provenance
The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.
| Stage | Outcome | Tool | Model | Prompt | Attempts | Completed |
|---|---|---|---|---|---|---|
| discover | success | OpenAlex-citations | — | — | 1 | 2026-06-19 |
| archive | success | semantic_scholar | — | — | 6 | 2026-06-25 |
| extract | success | cached | — | — | 2 | 2026-06-26 |
| clean | success | clean | — | — | 1 | 2026-06-19 |
| chunk | success | chunk | — | — | 1 | 2026-06-19 |
| embed | success | embed | Qwen/Qwen3-Embedding-8B | — | 1 | 2026-06-19 |
| promote | success | — | — | — | 1 | 2026-06-19 |
| summarize | success | llm | qwen3.6-27b-prismaquant | summ-v5 | 1 | 2026-06-26 |
| tag | success | vector_similarity | — | — | 6 | 2026-06-19 |
| verify | success | — | — | — | 1 | 2026-06-26 |
Summary generated by qwen3.6-27b-prismaquant on 2026-06-26; verification: verified.
Topics
Ranked by relevance to this paper. Hover a topic for its definition.
Information type
What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).
- Methodological Resource: validation psychometrics, tool software