On Assessing Driver Awareness of Situational Criticalities: Multi-modal Bio-Sensing and Vision-Based Analysis, Evaluations, and Insights

Siddharth, Siddharth; Trivedi, Mohan M. · 2020 · Crossref

DOI: 10.3390/brainsci10010046

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This study addresses the challenge of inferring driver awareness and differentiating between safe and critical driving situations using multi-modal bio-sensing and vision-based analysis. While computer vision is the standard for driver monitoring, it suffers from limitations such as occlusion, varying illumination, and a lack of personalization regarding individual physiological responses to fatigue or stress. The authors argue that wearable bio-sensors, specifically electroencephalogram (EEG), photoplethysmogram (PPG), and galvanic skin response (GSR), offer robust physiological insights that can complement vision systems. The research aims to evaluate the performance of these modalities individually and in fusion for classifying driver attention and detecting hazardous events, testing the hypothesis that low-temporal-resolution sensors like PPG and GSR can perform comparably to EEG and vision, and that feature fusion enhances classification accuracy. The experimental design involved twelve participants interacting with a driving simulator while viewing video stimuli from two datasets: the publicly available KITTI dataset and a newly collected LISA dataset featuring autonomous driving footage. Data was collected using a 14-channel Emotiv EEG headset, a Biovotion armband for PPG and GSR, and a frontal camera for facial expression analysis. The authors developed a novel feature extraction pipeline combining traditional signal processing and deep learning. For EEG, they calculated mutual information features to map brain region interactions and converted power spectrum density (PSD) heatmaps of theta, alpha, and beta bands into RGB images processed by a pre-trained VGG-16 network. For PPG and GSR, they extracted heart-rate variability and statistical time-domain features, alongside spectrogram images also processed by VGG-16. Facial features were derived from 49 localized landmarks and deep learning features from a VGG-Faces network. To capture temporal trends, particularly for high-resolution tasks, the authors employed Long Short-Term Memory (LSTM) networks on sequential EEG and vision features. The results demonstrate that the proposed methods outperform previously reported studies in classifying driver attention and detecting hazardous versus non-hazardous situations over short two-second intervals. The study highlights that EEG and vision data are most effective for high-resolution temporal classification due to their rapid response times, whereas PPG and GSR are better suited for longer time periods due to slower physiological reaction times. The fusion of features from different sensor modalities was shown to boost classification performance compared to using any single modality independently. The use of pre-trained deep neural networks allowed for effective feature extraction despite the limited scale of bio-sensing data, bridging the gap between laboratory-controlled experiments and real-world applicability. The significance of this work lies in its comprehensive evaluation of multiple sensing modalities for driver awareness, providing a scalable framework for real-time signal acquisition and classification. By validating the utility of compact, wireless bio-sensors in conjunction with vision systems, the study supports the development of more robust, personalized advanced driver assistance systems. The findings suggest that integrating physiological data can overcome the limitations of vision-only systems, particularly in detecting subtle changes in driver state that may not be visible through facial expressions alone, thereby enhancing safety in both manual and autonomous driving contexts.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-nvidia on 2026-08-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• drowsiness detection algorithms
• distraction detection algorithms
• situational awareness
• gaze based attention detection
• telematics crash prediction

Information type

What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).

• Empirical Findings: physiological data
• Methodological Resource: tool software, validation psychometrics