Concurrent talking in immersive virtual reality: on the dominance of visual speech cues

González-Franco, Mar; Maselli, Antonella; Florêncio, Dinei; Smolyanskiy, Nikolai; Zhang, Zhengyou · 2017 · OpenAlex-citations

DOI: 10.1038/s41598-017-04201-x

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This study investigates how visual speech cues, specifically lip movements, influence selective auditory attention in noisy environments with concurrent speakers. The research addresses the "cocktail party effect," where humans selectively listen to a target speaker while ignoring others, by examining the dominance of visual modality over auditory input in immersive Virtual Reality (VR). The authors hypothesize that asynchronous audiovisual speech (AVS) cues—where lip movements do not match the corresponding audio—can disrupt selective attention and increase comprehension errors. The researchers conducted two experiments using an Information Masking Task in a VR setup with spatialized audio. In the first experiment, 32 participants listened to two simultaneous speakers (a Target and a Mask) and performed tasks to identify a specific signal and recall a command. The study manipulated AVS congruency across three conditions: SYNC (synchronized audio and video), NOLIPS (audio only, static video), and ASYNC1 (Target’s lips matched the Mask’s audio). A second control experiment with 20 participants added an ASYNC2 condition, where the Target’s lips were unrelated to both speakers’ audio, to isolate the effects of specific multimodal interactions. Performance was measured using response times, accuracy, and Inverse Efficiency Scores (IES). The results demonstrated a significant dominance of visual speech cues in modulating selective attention. In the ASYNC1 condition, where the Target’s lips articulated the Mask’s speech, participants exhibited significantly higher error rates and lower efficiency in recalling the Target’s command compared to SYNC and NOLIPS conditions. Crucially, analysis of "mixed-up" responses revealed that participants frequently recalled information from the Mask speaker when the Target’s lips matched the Mask’s audio, indicating that visual cues effectively redirected auditory attention from the Target to the Mask. This disruption was not observed in the ASYNC2 condition, where lips were unrelated to the audio, suggesting that the interference was driven by the specific congruence between visual articulation and auditory content rather than general asynchrony. These findings imply that visual speech cues can hijack auditory selective attention through bottom-up multisensory integration, overriding top-down attentional processes. The study highlights that in immersive environments, mismatched AVS cues can severely impair speech comprehension by causing listeners to attend to the wrong speaker. This has significant implications for cognitive neuroscience, demonstrating the powerful role of vision in auditory perception, and for VR technology design, emphasizing the need for precise audiovisual synchronization to avoid disrupting user comprehension and attention in virtual communications.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-prismaquant on 2026-06-25; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• multisensory crossmodal