Reliable but multi-dimensional cognitive demand in operating partially automated vehicles
DOI: 10.1186/s41235-024-00591-5
archive: archived pipeline: cataloged verified
Abstract
The reliability of cognitive demand measures in controlled laboratory settings is well-documented; however, limited research has directly established their stability under real-life and high-stakes conditions, such as operating automated technology on actual highways. This study examined the reliability of five cognitive demand measures while participants operated partially automated vehicles on real roads across four occasions. Seventy-one participants (aged 18-64) drove on actual highways while their heart rate, heart rate variability, EEG alpha power, and behavioral performance on the Detection Response Task were measured simultaneously. EEG alpha power had excellent test-retest reliability, heart rate and its variability were good, and Detection Response Task reaction time and hit-rate had moderate reliabilities. Despite high reliability of each measure, low intercorrelations among measures were observed, and internal consistency was better when cognitive demand was estimated as a multi-factorial construct, suggesting they tap into different aspects of cognitive demand.
Summary
Open-access Brief Report in Cognitive Research: Principles and Implications testing the test-retest reliability of five cognitive demand measures while drivers operated four Level-2 partially automated vehicles (Cadillac CT6, Nissan Rogue, Tesla Model 3, Volvo XC90) on real highways across four occasions. Seventy-one participants drove each vehicle on a counterbalanced highway route in both partial automation and manual modes for at least 18 minutes per direction. ICC1s showed excellent reliability for EEG alpha power (~0.94-0.96), good reliability for heart rate (~0.80-0.87), moderate reliability for HRV/RMSSD (~0.72-0.77), and moderate reliability for DRT reaction time (~0.69-0.75) and DRT hit-rate (~0.62-0.63), with overlapping confidence intervals between automation and manual conditions. Intercorrelations among measures were low, omega hierarchical was poor (~0.34-0.36), but omega total supporting a multi-factor model was substantially higher (~0.67-0.73). Authors conclude these measures reliably capture cognitive demand in real-world automation research but tap distinct dimensions rather than a single general workload construct, motivating multimodal assessment.
Key finding
In real-world Level-2 automated highway driving, EEG alpha power shows excellent and cardiovascular measures show good test-retest reliability across four occasions, while DRT reaction time and hit-rate show moderate reliability; low intercorrelations and superior fit of a multi-factor (omega total) over a general-factor (omega hierarchical) model indicate cognitive demand is best treated as a multi-dimensional construct in applied automation research.
Methodology
Repeated-measures on-road study. Seventy-one drivers each completed four sessions, one per vehicle (Cadillac CT6, Nissan Rogue, Tesla Model 3, Volvo XC90), counterbalanced. Each session included approximately 18 minutes of partial automation and 18 minutes of manual driving on the same highway in counterbalanced directions, with a research assistant present. Continuous measures: ECG-derived heart rate (BPM) and HRV (RMSSD) sampled at 2000 Hz with bandpass filtering; EEG alpha power (8-12 Hz, FFT) following standard pipeline with mastoid reference and EOG monitoring; behavioral performance on the ISO 17488 Detection Response Task with a tactile actuator on the left bicep and a microswitch response (Red Scientific equipment). Test-retest reliability quantified via single-score intraclass correlations (ICC1) from one-way random-effects models. Internal consistency across measures evaluated with omega hierarchical, omega total, R1R, and Cronbach's alpha; analyses in R.
Sample size: N=71 drivers (25 female), aged 21-64 (M=40.32, SD=13.37); each tested across 4 vehicles.
Quality score: 5 / 5