Real-time control of a hearing instrument with EEG-based attention decoding

Hjortkjær, Jens; Wong, Daniel D.E.; Catania, Alessandro; Märcher-Rørsted, Jonatan; Ceolini, Enea; Fuglsang, Søren A.; Kiselev, Ilya; Di Liberto, Giovanni; Liu, Shih-Chii; Dau, Torsten; Slaney, Malcolm; de Cheveigné, Alain · 2024 · Crossref

DOI: 10.1101/2024.03.01.582668

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the persistent challenge of speech perception in noisy environments for hearing aid users, specifically the inability of current devices to determine which sound source a user intends to attend to. While speech separation technology can isolate individual voices, hearing aids require a mechanism to selectively enhance the target speaker and suppress interferers. The authors present a real-time brain-computer interface (BCI) system that enables "neurosteering," where a hearing instrument dynamically adjusts audio gains based on real-time decoding of the user’s auditory attention from electroencephalogram (EEG) signals. This closed-loop system aims to function as a neural prosthesis, automatically enhancing relevant speech streams in complex acoustic scenes. The system integrates a wireless multi-microphone hardware platform called WHISPER with an EEG-based attention decoder. WHISPER utilizes an ad hoc array of distributed microphones to perform low-latency spatial beamforming, separating mixed sound signals into distinct speech streams. The attention decoding module employs Canonical Correlation Analysis (CCA) to measure the synchronization between continuous EEG responses and the acoustic envelopes of the separated speech streams. Implemented using the OpenVibe software platform, the system preprocesses EEG and audio data, applies CCA to identify the attended stream based on correlation strength, and uses a linear classifier to determine attention state. The decoded attention signal then controls the relative gains of the audio channels, enhancing the attended speaker while suppressing others. The authors implemented two control strategies: a direct non-linear mapping of classification probabilities to gain, and a state-space model that accumulates information over time to reduce erratic gain changes. The study demonstrates the feasibility of this real-time closed-loop system through case studies involving listeners with hearing impairment. In these demonstrations, participants steered acoustic feedback of competing speech streams via real-time attention decoding. The system successfully identified attended speakers and adjusted audio levels accordingly. The authors note that offline decoding accuracies using their CCA pipeline reached approximately 70–80% with decoding windows of 6–8 seconds. The WHISPER platform achieved signal-to-distortion ratios of roughly 6 dB with four microphones and 9 dB with twelve microphones in reverberant conditions. The paper also explores the impact of visual inputs, showing that visual feedback of speakers' faces can assist in decoding attention when audibility is low. The significance of this work lies in its comprehensive integration of hardware and software components into a functional real-time prototype, moving beyond offline studies to demonstrate practical application. The authors discuss critical challenges for future deployment, including processing latency, user tolerance to errors, and the adaptability of elderly hearing-impaired users. By providing a publicly available software implementation, the study facilitates further research into cognitively controlled hearing technology, offering a pathway toward hearing aids that actively assist users in navigating complex acoustic environments through neural feedback.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-nvidia on 2026-08-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• auditory