Optimized driver fatigue detection method using multimodal neural networks
DOI: 10.1038/s41598-025-86709-1
archive: archived pipeline: cataloged verified
Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)
Summary
This study addresses the critical safety issue of driver fatigue, which significantly contributes to road accidents. While existing detection methods rely on either subjective self-reports or objective single-modal data (such as vehicle behavior, physiological signals, facial features, or voice), these approaches often suffer from limitations like interpretation bias, environmental interference, or insufficient capture of complex human physiological states. To overcome these challenges, the authors propose a comprehensive multimodal neural network approach that integrates facial data and physiological signals to provide precise and reliable fatigue detection. The research utilizes the DROZY dataset, which contains synchronized physiological and facial data collected from 14 participants under sleep deprivation conditions. The data includes electroencephalograms (EEG), electrocardiograms (ECG), electromyograms (EMG), electrooculograms (EOG), and near-infrared facial videos. The authors developed two distinct models: a multimodal feature combination model and a novel multimodal feature coupled model. In the coupled model, features extracted by a ResNet18 encoder (for facial images) and an LSTM encoder (for physiological time-series data) are not merely concatenated. Instead, they serve as mutual weights, dynamically influencing each other’s contribution to the final prediction through a coupling mechanism. This design captures complex interdependencies between modalities. To ensure robustness, the study employed a subject-based data splitting strategy and a majority voting decision mechanism to aggregate predictions from multiple classifiers. The results demonstrate that the multimodal feature coupled model significantly outperforms the feature combination model. The coupled model achieved an accuracy of 98.41%, with precision, recall, and F1-score all at approximately 98.4%. In contrast, the combination model achieved an accuracy of 94.87% and an F1-score of 95.00%. The superior performance of the coupled model is attributed to its ability to model the dynamic interaction between facial cues and physiological states, rather than treating them as independent inputs. The subject-based evaluation method ensured that the results reflected genuine generalization capabilities, avoiding bias from sample repetition. The significance of this work lies in its advancement of multimodal fusion techniques for driver monitoring systems. By demonstrating that dynamic feature coupling yields higher accuracy than traditional concatenation methods, the study provides a more effective framework for detecting fatigue. The integration of high-accuracy detection with a majority voting strategy offers a reliable solution for real-world applications, potentially enhancing road safety through efficient, automated monitoring systems in vehicles. This approach addresses the limitations of single-modal systems and offers a robust tool for mitigating fatigue-related accidents.
Key finding
Multimodal feature-coupled model achieved 98.41% accuracy in driver fatigue detection by dynamically linking features from EEG, ECG, and facial data as mutual weights, significantly outperforming simple multimodal concatenation (94.87% accuracy).
Methodology
lab_experiment
Sample size: 14
Provenance
The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via discover_direct_oa on 2026-05-03 (3 acquisition events logged).
| Stage | Outcome | Tool | Model | Prompt | Attempts | Completed |
|---|---|---|---|---|---|---|
| discover | success | — | — | — | 1 | 2026-05-03 |
| archive | success | — | — | — | 1 | 2026-05-03 |
| extract | success | cached | — | — | 3 | 2026-06-07 |
| clean | success | — | — | — | 1 | 2026-06-01 |
| chunk | success | — | — | — | 1 | 2026-06-01 |
| embed | success | — | — | — | 1 | 2026-06-02 |
| enrich | success | semantic_scholar | — | — | 2 | 2026-06-15 |
| promote | success | — | — | — | 1 | 2026-05-03 |
| summarize | success | llm | qwen3.6-27b-prismaquant | summ-v5 | 3 | 2026-06-07 |
| tag | success | vector_similarity | — | — | 16 | 2026-06-11 |
| verify | success | — | — | — | 1 | 2026-05-08 |
Summary generated by qwen3.6-27b-prismaquant on 2026-06-07; verification: verified.
Topics
Ranked by relevance to this paper. Hover a topic for its definition.
Information type
What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).
- Empirical Findings: physiological data
- Methodological Resource: tool software, validation psychometrics