Reproducible machine learning research in mental workload classification using EEG
DOI: 10.3389/fnrgo.2024.1346794
archive: archived pipeline: cataloged
Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)
Summary
This paper addresses the critical lack of reproducibility in machine learning (ML) research using electroencephalography (EEG) for mental workload classification. Motivated by the broader reproducibility crisis in science—where a significant majority of researchers cannot replicate others’ experiments—and the specific challenges in neuroergonomics, such as the difficulty of generalizing EEG-based models across subjects, sessions, and contexts, the authors aim to establish standardized guidelines for reproducible EEG-ML pipelines. The study focuses on mental workload recognition as a representative sub-area of passive brain-computer interfaces, where ML is essential for decoding user states in real-time applications. The methodology involves a two-stage systematic literature review and the development of a structured checklist. First, the authors reviewed existing reproducibility efforts in both ML and EEG domains, noting that while separate guidelines exist for each field, an integrated framework was missing. They adopted the Cross-Industry Standard Process for Data Mining (CRISP-DM) framework to structure their new guidelines, aligning reproducibility requirements with the six standard phases of data mining: Business Understanding, Data Understanding, Data Preparation, Modeling, Evaluation, and Deployment. Second, they conducted a systematic search across Scopus, Web of Science, ACM Digital Library, and PubMed for studies on EEG-based mental workload reproducibility. From 51 unique articles identified, only 13 met the eligibility criteria of focusing on EEG, mental workload, and reproducibility or generalizability. The authors then evaluated these studies against their proposed checklist, which emphasizes full (R1) reproducibility, requiring the sharing of code, data, and detailed experimental documentation. The findings reveal significant gaps in current research practices. The systematic review highlighted that few studies explicitly address reproducibility in this domain. Among the relevant studies, reproducibility was demonstrated through various means, such as testing across different electrode configurations, 2D/3D environments, larger participant pools, different tasks, and over time. However, the evaluation against the new guidelines identified major limitations in the broader literature: inadequate reporting of performance on unseen test data, lack of open sharing of raw or preprocessed data and code, and insufficient documentation of computational resources and environment properties necessary for training and inference. The paper concludes that while some studies show promise in generalizability, the field lacks systematic transparency. The significance of this work lies in the provision of a concrete, actionable checklist for researchers to enhance the reliability and usability of EEG-ML techniques. By integrating ML best practices with EEG-specific standards (such as BIDS and COBIDAS recommendations), the guidelines aim to foster a more collaborative research environment. This structured approach not only improves the credibility of individual studies but also facilitates the development of robust, generalizable applications for monitoring and supporting professionals in high-focus work environments. The framework is designed to be adaptable to other EEG-based mental state assessment domains, promoting long-term scientific progress and seamless deployment of neuroergonomic applications.
Provenance
The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.
| Stage | Outcome | Tool | Model | Prompt | Attempts | Completed |
|---|---|---|---|---|---|---|
| discover | success | Crossref | — | — | 1 | 2026-08-09 |
| archive | success | canonical_url | — | — | 1 | 2026-08-09 |
| extract | success | cached | — | — | 5 | 2026-08-23 |
| clean | success | clean | — | — | 2 | 2026-08-10 |
| chunk | success | chunk | — | — | 2 | 2026-08-10 |
| embed | success | embed | Qwen/Qwen3-Embedding-8B | — | 2 | 2026-08-10 |
| promote | success | — | — | — | 1 | 2026-08-09 |
| summarize | success | llm | qwen3.8-27b-gittensor | summ-v5 | 3 | 2026-08-23 |
| tag | success | vector_similarity | — | — | 17 | 2026-08-11 |
| verify | success | — | — | — | 2 | 2026-08-09 |
Summary generated by qwen3.8-27b-gittensor on 2026-08-23; verification: pending re-verification.
Topics
Ranked by relevance to this paper. Hover a topic for its definition.
Information type
What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).
- Empirical Findings: physiological data