EEG-fNIRS-based hybrid image construction and classification using CNN-LSTM

Mughal, Nabeeha Ehsan; Khan, Muhammad Jawad; Khalil, Khurram; Javed, Kashif; Sajid, Hasan; Naseer, Noman; Ghafoor, Usman; Hong, Keum-Shik · 2022 · Crossref

DOI: 10.3389/fnbot.2022.873239

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This study addresses the challenge of integrating electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) data for hybrid brain-computer interface (BCI) applications. While hybrid systems combine the high temporal resolution of EEG with the superior spatial resolution of fNIRS, significant discrepancies in sampling rates and channel counts typically necessitate downsampling or channel selection, leading to valuable information loss. To overcome this, the authors propose a novel method that constructs hybrid images from raw signals using recurrence plots (RPs) without downsampling, enabling the preservation of full signal information. The research aims to improve classification accuracy for cognitive states by leveraging a deep learning architecture capable of processing these non-linear, time-dependent features. The methodology utilizes an open-source dataset from Technische Universität Berlin, comprising simultaneous EEG and fNIRS recordings from 26 healthy participants performing n-back cognitive tasks. EEG data were recorded at 1,000 Hz using 30 electrodes, while fNIRS data were captured at 10.4 Hz using 16 optodes. After preprocessing, including bandpass filtering for EEG and hemoglobin conversion for fNIRS, the time-series data were transformed into 2D recurrence plots. These RPs served as image inputs for a custom time-distributed convolutional neural network and long short-term memory (CNN-LSTM) model. The CNN layers extracted spatial features from the RPs, while the LSTM layers captured chronological dependencies and time-series dynamics. The model was trained to classify four states: 0-back, 2-back, 3-back, and rest, using 10-fold cross-validation to ensure robust generalization. The results demonstrate that the proposed RP-based CNN-LSTM approach significantly outperforms traditional methods that rely on data reduction. The model achieved average classification accuracies of 78.44% for fNIRS alone, 86.24% for EEG alone, and 88.41% for the hybrid EEG-fNIRS system. Maximum accuracies reached 85.9%, 88.1%, and 92.4% for fNIRS, EEG, and hybrid modalities, respectively. These findings confirm that constructing hybrid images via recurrence plots allows for the effective integration of disparate neuroimaging modalities without the information loss associated with downsampling. The significance of this work lies in its contribution to robust, real-time BCI systems. By eliminating the need for aggressive data reduction, the method preserves critical neural dynamics, enhancing the reliability of brain state detection outside controlled laboratory environments. The successful application of time-distributed layers in this context offers a viable pathway for developing more accurate hybrid BCIs for neurorehabilitation, human-machine interaction, and monitoring cognitive states such as mental workload and fatigue.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-nvidia on 2026-08-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• neuro workload indices