Separation Management: Automation Reliability Meta-Analysis and Conflict Probe Reliability Analysis
archive: archived pipeline: cataloged verified
Get this paper ↗ (full text — opens at the source; we link to it, we don't host it)
Summary
This study addresses the need to establish a valid performance criterion for the En Route Automation Modernization (ERAM) Conflict Probe, specifically regarding its conflict-detection accuracy. The research was motivated by the desire to determine if the Conflict Probe’s accuracy is sufficient to justify displaying alerts on the radar side (R-side) of controller workstations, thereby improving situation awareness. The core problem was that existing literature lacked a definitive threshold for automation accuracy required to improve joint human-automation performance, and the Conflict Probe’s accuracy varied significantly depending on the analysis technique used. The researchers employed a two-part methodology. First, they conducted a meta-analysis of 12 studies from the human factors literature to determine the relationship between automation performance metrics and system performance relative to a non-automated baseline. They analyzed four metrics: reliability (overall percent correct), hit rate, false alarm rate, and Positive Predictive Value (PPV). Unlike previous informal analyses, this study used Cohen’s *d* effect sizes to measure relative performance, providing a continuous variable less susceptible to sample size distortions. Second, they analyzed engineering data from FAA Concept Analysis studies to compute the Conflict Probe’s accuracy using these same metrics. This allowed for a direct comparison between the empirically derived criteria from the literature and the observed performance of the ERAM system. The meta-analysis revealed a statistically significant positive relationship between automation reliability and system performance, identifying a "crossover point" of 65% reliability. Automation exceeding this threshold was likely to improve performance, though the estimate carried a broad 95% confidence interval (0.39 to 0.72) due to data variability. No statistically reliable relationships were found for hit rate, false alarm rate, or PPV. When evaluating the Conflict Probe, results depended heavily on how correct rejections were counted. Under liberal criteria including all correct rejections, the probe’s reliability exceeded the upper bound of the confidence interval, suggesting it could improve Air Traffic Control performance. However, under strict criteria excluding correct rejections, the reliability fell below the lower bound. While PPV showed large improvements over baseline, the meta-analysis could not establish a definitive PPV cutoff criterion. The study concludes that while the analyses provide insight, they do not definitively establish the Conflict Probe’s operational acceptability or a clear accuracy criterion. The authors emphasize that operational input is essential to determine which aircraft encounters should be included in test sets and to define a justifiable accuracy threshold specific to air traffic control. They recommend further study to examine whether false alarms are operationally acceptable and to identify how different accuracy metrics impact controller performance in the specific context of conflict detection. The findings serve as input for designing future evaluations to derive operationally meaningful metrics for automation acceptance.
Key finding
The meta-analysis identified a 65% reliability threshold for automation to improve performance, but the Conflict Probe's acceptability depended on whether correct rejections were included in the accuracy calculation.
Methodology
meta_analysis
Provenance
The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via bulk_ingest_rosap on 2026-05-23 (6 acquisition events logged).
| Stage | Outcome | Tool | Model | Prompt | Attempts | Completed |
|---|---|---|---|---|---|---|
| discover | success | rosap | — | — | 2 | 2026-05-23 |
| archive | success | — | — | — | 1 | 2026-05-23 |
| extract | success | cached | — | — | 2 | 2026-06-10 |
| clean | success | — | — | — | 1 | 2026-06-01 |
| chunk | success | — | — | — | 1 | 2026-06-01 |
| embed | success | — | — | — | 1 | 2026-06-02 |
| enrich | success | — | — | — | 1 | 2026-05-23 |
| promote | success | — | — | — | 1 | 2026-05-23 |
| summarize | success | llm | qwen3.6-27b-prismaquant | summ-v5 | 3 | 2026-06-10 |
| tag | success | vector_similarity | — | — | 24 | 2026-06-11 |
| verify | success | — | — | — | 2 | 2026-06-10 |
Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.
Topics
Ranked by relevance to this paper. Hover a topic for its definition.
Information type
What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).
- Methodological Resource: validation psychometrics