Probability of Detection (POD)-based Metric for Evaluation of Classifiers Used in Driving Behavior Prediction
DOI: 10.36001/phmconf.2019.v11i1.774
archive: archived pipeline: cataloged verified
Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)
Summary
This paper addresses the limitations of current classifier certification methods, specifically the Receiver Operator Curve (ROC), which fails to account for process parameters such as prediction time. To overcome this, the authors propose a new evaluation metric based on Probability of Detection (POD), a reliability measure established in Nondestructive Testing (NDT) and Structural Health Monitoring (SHM). The study applies this POD-based approach to evaluate binary classifiers used in driving behavior prediction, aiming to quantify the reliability of machine learning models in autonomous and assisted driving systems. The research utilizes data from a professional driving simulator (SCANeRTM studio) involving three driving maneuvers: lane changing to the right, lane keeping, and lane changing to the left. Three classifiers are compared: Fuzzy Logic-based Hidden Markov Models (FL-HMM), Support Vector Machines (SVM), and Artificial Neural Networks (ANN). The authors implement two POD assessment approaches: the signal-response method, which analyzes the linear relationship between detection rate and a process parameter (prediction time), and the hit/miss method, which uses Generalized Linear Models (GLM) to model binary outcomes. For the signal-response analysis, detection rates were calculated at 120 time points (every 0.05 seconds) from six seconds before the maneuver to the time of execution. Confidence bounds were established using the Wald method and Fisher’s information matrix to account for data uncertainty. The results demonstrate that the POD metric effectively differentiates classifier performance based on prediction time. In the signal-response comparison, regression analysis was performed on the detection rates of FL-HMM, SVM, and ANN. The authors selected the model that best fit linearity and uniform variance criteria, deriving specific intercepts and slopes for the POD curves. The study successfully generated POD curves for varying time parameters, allowing for a probabilistic assessment of when each classifier reliably detects a driving maneuver. Additionally, the hit/miss approach was applied to FL-HMM to predict upcoming maneuvers, establishing confidence bounds for the binary classification outcomes. The significance of this work lies in introducing a standardized, reliability-based certification tool for machine learning classifiers in safety-critical applications. By incorporating process parameters like prediction time, the POD metric provides a more comprehensive evaluation than ROC alone. This approach enables the quantification of classifier reliability with specific confidence levels, such as a 90% probability of detection at a 95% confidence level. The findings suggest that POD-based metrics can serve as a robust standard for selecting and certifying classifiers in driver assistance systems, ensuring that models meet rigorous reliability requirements before deployment.
Key finding
The proposed Probability of Detection (POD) metric successfully evaluates and compares the reliability of FL-HMM, SVM, and ANN classifiers for driving behavior prediction by incorporating process parameters like prediction time, offering a more comprehensive assessment than traditional ROC curves.
Methodology
simulation_modeling
Provenance
The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via author_sweep_intake on 2026-05-28.
| Stage | Outcome | Tool | Model | Prompt | Attempts | Completed |
|---|---|---|---|---|---|---|
| discover | success | author_sweep | — | — | 2 | 2026-05-28 |
| archive | success | canonical_url | — | — | 1 | 2026-06-04 |
| extract | success | cached | — | — | 3 | 2026-06-10 |
| clean | success | clean | — | — | 1 | 2026-06-04 |
| chunk | success | chunk | — | — | 1 | 2026-06-04 |
| embed | success | embed | Qwen/Qwen3-Embedding-8B | — | 1 | 2026-06-04 |
| enrich | success | — | — | — | 1 | 2026-05-28 |
| promote | success | — | — | — | 1 | 2026-06-04 |
| summarize | success | llm | qwen3.6-27b-prismaquant | summ-v5 | 2 | 2026-06-10 |
| tag | success | vector_similarity | — | — | 15 | 2026-06-11 |
| verify | success | — | — | — | 2 | 2026-06-10 |
Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.
Topics
Ranked by relevance to this paper. Hover a topic for its definition.
Information type
What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).
- Methodological Resource: validation psychometrics, tool software
- Theoretical Contribution: computational model