The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence

Marucci, Matteo; Di Flumeri, Gianluca; Borghini, Gianluca; Sciaraffa, Nicolina; Scandola, Michele; Pavone, Enea Francesco; Babiloni, Fabio; Betti, Viviana; Aricò, Pietro · 2021 · Crossref

DOI: 10.1038/s41598-021-84196-8

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This study investigates how multisensory integration and perceptual load influence target detection performance, mental workload, and the sense of presence in virtual reality (VR). Motivated by the observation that real-world experiences are multimodal and that multisensory facilitation is modulated by the amount of information processed (perceptual load), the researchers aimed to determine if auditory and vibrotactile stimuli enhance attentional capture more effectively under high perceptual load conditions. The study utilized a realistic VR driving scenario to bridge the gap between controlled laboratory experiments and naturalistic environments, allowing for the simultaneous assessment of behavioral, subjective, and neurophysiological responses. The experimental design involved eighteen healthy male participants who performed a target detection task while driving a virtual car on a racetrack. The task required detecting and hitting sphere-like targets appearing on the left or right side of the track. Two levels of perceptual load were manipulated: a low load condition with high visibility and sunny weather, and a high load condition featuring mist, rain, thunder, and a higher frequency of targets. Four sensory conditions were tested: visual only (V), visual-auditory (VA), visual-tactile (VT), and trimodal visual-auditory-tactile (VAT). Data collection included behavioral performance metrics, subjective questionnaires for workload (NASA TLX) and presence, and physiological recordings via Electroencephalography (EEG) and Galvanic Skin Response (GSR). EEG analysis focused on Event-Related Potentials (specifically P300 amplitude and latency) and frequency-domain features (theta and alpha bands) to assess neural processing and mental workload. The results demonstrated that multisensory stimuli significantly improved target detection performance only under high perceptual load conditions compared to visual stimulation alone. Neurophysiologically, multisensory stimulation decreased EEG-based workload indices. However, subjective perceived workload, measured via the NASA TLX, was reduced only in the trimodal (VAT) condition. The trimodal stimulation also proved most effective in enhancing the sense of presence compared to bimodal or unimodal conditions. GSR components were significantly higher in the high load condition, indicating increased arousal. Furthermore, multimodal stimulations involving auditory input (VA and VAT) induced a significant decrease in P300 latency and an increase in P300 amplitude compared to unimodal or visual-tactile conditions. These ERP changes suggest that the inclusion of auditory stimulation facilitates faster and more effective neural processing and detection of environmental stimuli. The findings provide significant insights into the relationship between multisensory integration and human cognition, particularly in complex, realistic environments. The study confirms that the benefits of multisensory cues are context-dependent, becoming critical for performance enhancement when perceptual load is high. Additionally, the results highlight the utility of VR in studying naturalistic multisensory processing and suggest that trimodal feedback is optimal for reducing subjective workload and enhancing immersion. These conclusions have implications for the design of human-machine interfaces and VR systems, indicating that incorporating auditory and tactile cues can optimize user performance and comfort in high-demand scenarios.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-nvidia on 2026-08-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• workload measurement
• multimodal feedback

Information type

What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).

• Empirical Findings: physiological data, self report data
• Theoretical Contribution: theory or model