Beyond the Wizard of Oz: Negative Effects of Imperfect Machine Learning to Examine the Impact of Reliability of Augmented Reality Cues on Visual Search Performance

Raikwar, Aditya; Mifsud, Domenick; Wickens, Christopher D.; Batmaz, Anil Ufuk; Warden, Amelia C.; Kelley, Brendan; Clegg, Benjamin A.; Ortega, Francisco R. · 2024 · IEEE Transactions on Visualization and Computer Graphics

DOI: 10.1109/tvcg.2024.3372062

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This study investigates the impact of imperfect machine learning (ML) cues on visual search performance in Augmented Reality (AR) environments, specifically addressing the phenomenon of automation bias. While AR cues can enhance search efficiency, previous research relied on Wizard-of-Oz (WoZ) methodologies where errors were manually simulated. The authors argue that ML systems generate errors differently than humans, potentially leading to distinct user behaviors. The research aims to determine if real-time ML-driven cues produce different automation bias effects compared to WoZ studies and to quantify the performance costs associated with imperfect automation. The experimental design involved 53 participants divided into two groups: one receiving 100% accurate visual cues (perfect automation) and the other receiving cues generated by a YOLOv5-Nano ML model with 88.9% accuracy (imperfect automation). Both groups also completed a control condition with no cues. Participants used Varjo XR-3 AR headsets to locate specific objects in a 3D environment. The ML system was trained on a synthetic dataset of 40,000 images to ensure realistic error patterns, such as confusion due to lighting or background complexity. An arrow cue served as the visual guide, pointing to the predicted target location. The study employed a mixed-subject design, comparing performance across cue reliability levels (between-subjects) and cue presence (within-subjects). The results demonstrated that while cueing generally improved accuracy and reduced search times compared to the no-cue condition, the reliability of the cue significantly influenced outcomes. Participants in the perfect cue condition achieved 98.7% accuracy, whereas those in the imperfect cue condition achieved only 90.7%. Crucially, performance with imperfect automation was significantly worse than with perfect automation. The study found strong evidence of automation bias, as participants frequently followed incorrect ML cues, leading to higher error rates and longer search times than in the uncued control condition. This indicates that users over-relied on the ML system, failing to adequately verify suggestions even when instructed that the system was not 100% reliable. The findings highlight the detrimental effects of imperfect automation in AR systems, showing that ML-driven cues can induce higher automation bias than previously observed in WoZ studies. The research underscores the importance of considering cue reliability in system design, as imperfect cues can degrade performance below baseline levels due to user over-trust. By providing open-source code and datasets, the authors facilitate further research into human-automation interaction, emphasizing the need for strategies to mitigate automation bias in safety-critical applications relying on AI-generated visual guidance.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-prismaquant on 2026-06-09; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• visual search