Perceived Mental Workload Classification Using Intermediate Fusion Multimodal Deep Learning
DOI: 10.3389/fnhum.2020.609096
archive: archived pipeline: cataloged verified
Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)
Summary
This study addresses the challenge of classifying perceived mental workload (PMWL) by leveraging multimodal physiological data through deep learning. While previous research has utilized various bio-signals for MWL detection, most studies rely on single modalities, ignoring the potential benefits of cross-modality information. The authors aimed to develop a flexible, modular deep neural network (DNN) capable of intermediate fusion, allowing for feature sharing between different data streams. The primary motivation was to create a system that adheres to principles of modularity and generalisability, enabling easy addition or removal of sensors without major structural changes to the model. The experimental design involved 22 participants (one excluded due to poor data quality) solving verbal logic puzzles, known as zebra puzzles, across five difficulty levels. Participants rated the difficulty of each puzzle on a seven-point scale, providing ground-truth labels for PMWL. Four physiological modalities were recorded simultaneously using LabStreamingLayer (LSL) for synchronization: functional near-infrared spectroscopy (fNIRS) for brain activity, galvanic skin response (GSR) for sympathetic arousal, photoplethysmography (PPG) for heart rate, and eye tracking (ET) for visual attention. The researchers designed an Intermediate Fusion Multimodal Network (IFMMoN), consisting of separate modality-specific networks (MNets) that feed into a shared Head network. Two architectural approaches were evaluated: one based on literature-specific layer configurations (e.g., CNNs for ET, LSTM-CNN for GSR) and one using only densely connected layers. Hyperparameter optimization was performed using Optuna. The results demonstrated that the proposed IFMMoN achieved high accuracy in classifying PMWL. Specifically, the model classified workload within-level accurate (0.985 levels) on the seven-point scale. The study confirmed that multimodal fusion significantly outperformed single-modality approaches, validating the hypothesis that combining diverse physiological signals enhances classification performance. Furthermore, the modular design proved effective, allowing for the integration of disparate data types with varying sampling rates and dimensions. The authors also found that the model’s performance was robust across different labeling schemes, including individual-specific and group-averaged difficulty ratings. The significance of this work lies in its demonstration that intermediate fusion multimodal deep learning can effectively classify PMWL with high precision. By prioritizing modularity and generalisability, the proposed architecture offers a scalable solution for brain-computer interfaces and human factors applications. The ability to dynamically add or remove modalities without retraining the entire network structure addresses a common limitation in existing multimodal systems. Additionally, the open availability of the dataset and code facilitates further research into multimodal signal processing and mental workload assessment. This approach provides a foundation for real-time, adaptive systems that can monitor cognitive load implicitly, reducing the need for intrusive self-assessment methods.
Provenance
The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.
| Stage | Outcome | Tool | Model | Prompt | Attempts | Completed |
|---|---|---|---|---|---|---|
| discover | success | Crossref | — | — | 1 | 2026-08-09 |
| archive | success | canonical_url | — | — | 1 | 2026-08-09 |
| extract | success | cached | — | — | 3 | 2026-08-10 |
| clean | success | clean | — | — | 1 | 2026-08-09 |
| chunk | success | chunk | — | — | 1 | 2026-08-09 |
| embed | success | embed | Qwen/Qwen3-Embedding-8B | — | 1 | 2026-08-09 |
| promote | success | — | — | — | 1 | 2026-08-09 |
| summarize | success | llm | qwen3.6-27b-nvidia | summ-v5 | 2 | 2026-08-10 |
| tag | success | vector_similarity | — | — | 11 | 2026-08-11 |
| verify | success | — | — | — | 2 | 2026-08-10 |
Summary generated by qwen3.6-27b-nvidia on 2026-08-10; verification: verified.
Topics
Ranked by relevance to this paper. Hover a topic for its definition.
Information type
What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).
- Empirical Findings: physiological data