Capsule Attention for Multimodal EEG-EOG Representation Learning With Application to Driver Vigilance Estimation

Zhang, Guangyi; Etemad, Ali · 2021 · Crossref

DOI: 10.1109/tnsre.2021.3089594

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the challenge of estimating driver vigilance to enhance transportation safety, specifically targeting issues arising from fatigue and distraction. The authors propose a novel multimodal deep learning architecture that fuses Electroencephalogram (EEG) and Electrooculogram (EOG) signals. While previous methods often struggled with the noise, artifacts, and complementary nature of biological signals, this work introduces a capsule attention mechanism to improve representation learning. The primary motivation is to create a system that can adaptively focus on salient temporal features and modalities, thereby learning robust hierarchical dependencies that are less susceptible to the uncertainties inherent in wearable sensor data. The proposed method employs a deep Long Short-Term Memory (LSTM) network to capture temporal dependencies in the data, followed by a capsule attention mechanism. This mechanism consists of lower-level and higher-level capsule layers that utilize dynamic routing to establish part-whole relationships. The lower-level capsules extract local features, while higher-level capsules aggregate this information to form global representations, allowing the model to selectively attend to relevant data segments. The input data undergoes preprocessing, including downsampling to 200 Hz and filtering, followed by feature extraction. EEG features are derived from Short-time Fourier Transform outputs, calculating Power Spectral Density and Differential Entropy. EOG features are extracted from eye movements such as blinks, saccades, and fixations. These features are fused and processed through the LSTM and capsule layers, with a final regression layer predicting vigilance scores. Experiments were conducted using the SEED-VIG dataset, which contains EEG and EOG recordings from 23 participants engaged in simulated driving. The study evaluated the model using both intra-participant and cross-participant validation schemes, measuring performance via Root Mean Square Error (RMSE) and Pearson Correlation Coefficient (PCC). The results demonstrate that the proposed capsule attention model significantly outperforms existing state-of-the-art solutions and baseline techniques, including unimodal EEG or EOG models and other multimodal approaches. The model achieved lower RMSE and higher PCC values, indicating superior accuracy in vigilance estimation. Furthermore, ablation studies and noise robustness analyses confirmed that the capsule attention mechanism enhances the separability of learned representations and provides greater resilience to signal noise compared to traditional attention mechanisms or standard deep learning architectures. The significance of this work lies in its demonstration that capsule networks can effectively handle the hierarchical and noisy nature of multimodal biological signals. By integrating capsule attention with LSTM networks, the authors provide a more robust framework for real-time driver monitoring systems. The findings suggest that leveraging part-whole relationships through dynamic routing allows for better discrimination of vigilance states, even in the presence of artifacts. This approach sets a new benchmark for multimodal EEG-EOG analysis, offering a promising direction for developing reliable, wearable brain-computer interfaces for safety-critical applications like autonomous driving assistance and fatigue detection.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-nvidia on 2026-08-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• drowsiness detection algorithms
• distraction detection algorithms
• gaze based attention detection

Information type

What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).

• Empirical Findings: physiological data
• Methodological Resource: tool software