Using Sensitivity and Bias in Signal Detection Theory to Predict Proportion Correctness: Simulation Study

Huang, Chunxi; He, Dengbo · 2023 · Proceedings of the Human Factors and Ergonomics Society Annual Meeting

DOI: 10.1177/21695067231193662

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This study addresses the limitations of using proportion correctness (PC) to evaluate drivers’ mental models of advanced driver assistance systems (ADAS). While PC is widely used, it is biased by the ratio of signal (existing ADAS functions) to noise (non-existing functions) in survey questions. Signal Detection Theory (SDT) offers unbiased metrics—sensitivity ($d'$) and response bias ($c$)—but lacks a closed-form solution to relate these metrics to PC. The authors aimed to derive an empirical equation linking $d'$, $c$, and PC to facilitate cross-study comparisons and meta-analyses. The researchers employed a two-part methodology involving numerical simulations and a real-world survey case study. First, they generated simulated data across 90 conditions, varying the number of questions ($N$), participants ($P$), and noise probabilities. They modeled both binary (yes/no) and rating-scale (1–6 Likert) responses, calculating $d'$ and $c$ using traditional and fuzzy SDT, respectively. By fitting 18,000 linear regression models, they extracted coefficients for an equation predicting PC from $d'$ and $c$. Second, they validated this equation using survey data from 287 participants who rated 49 statements regarding ADAS functions. The study compared the PC predicted by the empirical equation against the PC calculated directly from participant responses. The results yielded a robust empirical equation: $PC = 0.5 + 0.2 \times d' + \beta_3 \times c$, where $\beta_3$ is a function of the signal-to-noise ratio (SN-ratio). The simulation showed that the coefficient for $d'$ consistently converged to 0.2, while the intercept stabilized at 0.5. The coefficient for bias ($\beta_3$) varied based on the SN-ratio: it was positive when signals were fewer than noises (SN-ratio < 1), negative when signals exceeded noises (SN-ratio > 1), and zero when they were equal. Validation against the survey data demonstrated high accuracy, with $R^2$ values of 0.82 for binary questions and 0.89 for rating questions. The significance of this work lies in providing a standardized method to translate SDT metrics into PC scores, enabling consistent comparisons across studies with different questionnaire designs. The findings reveal that response bias impacts accuracy differently depending on question composition; for instance, a "yes" bias improves scores only when most statements describe existing functions. The authors conclude that to obtain an objective measure of knowledge independent of bias, researchers should design surveys with an SN-ratio of 1. This approach allows PC to be predicted solely by sensitivity ($d'$), simplifying the evaluation of driver understanding and supporting future meta-analyses in human factors research.

Key finding

An empirical equation derived from simulations accurately predicts proportion correctness from signal detection theory metrics, achieving R-squared values greater than 0.8 when validated against survey data.

Methodology

simulation_modeling

Sample size: 287

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• signal detection theory

Information type

What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).

• Methodological Resource: validation psychometrics
• Theoretical Contribution: computational model