Multimodal System to Detect Driver Fatigue Using EEG, Gyroscope, and Image Processing

Karuppusamy, Naveen Senniappan; Kang, Bo-Yeong · 2020 · Crossref

DOI: 10.1109/access.2020.3009226

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the critical safety issue of driver fatigue, a major contributor to traffic accidents, by proposing a hybrid multimodal detection system. While existing methods rely on single modalities such as electroencephalography (EEG), vehicle dynamics, or image processing, each approach has distinct limitations regarding accuracy and environmental sensitivity. To overcome these constraints, the authors developed a neural network-based system that fuses data from three sources: EEG signals for physiological state, gyroscope data for head motion, and image processing for facial behavioral signs. The study aims to demonstrate that a multimodal approach yields superior detection performance compared to individual sensor systems. The experimental design involved five licensed drivers aged 21–30 who participated in simulated driving sessions. Data was collected under two primary conditions: a normal driving state and a fatigue state, induced by sleep deprivation (less than six hours of sleep). Five specific scenarios were recorded to capture various fatigue indicators, including normal driving, yawning, head nodding, extreme drowsiness, and falling asleep. Raw data from EEG headsets, gyroscopes, and cameras were processed individually through three separate neural network modules: an EEG Module (EM), a Gyroscope Module (GM), and a Vision Module (VM). The outputs from these modules served as inputs for a final Deep Neural Network (DNN). This DNN utilized a Recurrent Neural Network (RNN) with 400 Long Short-Term Memory (LSTM) cells to handle time-series data, followed by a neural network with three hidden layers using ReLU activation and dropout regularization. The system was implemented using Google’s TensorFlow library. The results indicated that the proposed multimodal system achieved a detection accuracy of 93.91% in identifying the drowsiness state of the driver. The study found that the hybrid system performed better than individual systems, effectively complementing the limitations of each modality. For instance, while EEG provides robust physiological data, it is complemented by behavioral signs detected via gyroscope and vision modules, which capture involuntary actions like yawning and head nodding. The DNN architecture, specifically the use of LSTM cells to process historical time instances, allowed for sensitive and accurate classification of six different physiological and behavioral states. The authors concluded that combining EEG, gyroscope, and image processing data through a deep learning framework provides a more reliable and robust method for real-time driver fatigue detection, potentially enhancing safety in intelligent transportation systems.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-nvidia on 2026-08-10; verification: verified_with_issues.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• drowsiness detection algorithms
• drowsiness
• distraction detection algorithms

Information type

What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).

• Empirical Findings: physiological data
• Methodological Resource: tool software, validation psychometrics