Transparent systems, opaque results: a study on automation compliance and task performance

Pharmer, Rebecca L.; Wickens, Christopher D.; Clegg, Benjamin A. · 2025 · Cognitive Research Principles and Implications

DOI: 10.1186/s41235-025-00619-4

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This study investigates how different forms of automation transparency influence human compliance and task performance in a simulated nautical collision avoidance scenario. Motivated by the increasing prevalence of automated decision support tools in safety-critical domains, the authors aim to determine which transparency features help operators calibrate their dependence on imperfect automation. The research is grounded in a model of human–automation team performance, focusing on automation reliability, task difficulty, and transparency. Specifically, the study tests whether explaining the algorithm’s logic (Experiment 1) or providing confidence estimates (Experiment 2) improves compliance and safety outcomes. The researchers conducted two experiments using a simulated task where participants controlled a ship to avoid collisions with oncoming vessels. The automated aid, "Caskade," provided maneuver recommendations based on safety, efficiency, and adherence to collision regulations, with an 87.5% reliability rate. In Experiment 1, participants received pre-task instructions explaining that the aid considered collision regulations in its recommendations. In Experiment 2, transparency was manipulated by displaying confidence estimates alongside recommendations, and task difficulty was explicitly categorized. Participants’ compliance with the aid’s advice and their resulting miss distances (safety metric) were recorded, alongside self-reported trust measures. Experiment 1 found that providing transparency regarding the algorithm’s consideration of collision regulations significantly increased compliance, particularly when the aid recommended maneuvers that violated standard heuristics. This calibrated compliance led to better safety outcomes, with larger miss distances compared to prior studies lacking this transparency. However, Experiment 2 revealed that displaying confidence estimates provided no benefits for compliance or safety. Furthermore, contrary to the hypothesis that difficult tasks would increase reliance on automation, participants showed lower compliance on difficult problems. This reduction in compliance was mediated by a loss of trust due to the salience of automation errors in complex scenarios. Additionally, both experiments demonstrated low correlations between self-reported trust and actual behavioral compliance, suggesting these constructs are influenced by different factors. The findings imply that not all forms of transparency are equally effective; explaining the procedural logic of an algorithm can improve calibrated compliance, while confidence ratings may not. The study highlights a critical interaction between task difficulty and trust, where increased difficulty can erode trust and reduce dependence on automation, even when the aid remains more reliable than the human operator. These results provide a framework for designing automation systems that account for how transparency, reliability, and task difficulty jointly influence human behavior, emphasizing the need for careful implementation of transparency features to ensure optimal human–automation team performance.

Key finding

Providing pre-task transparency about the algorithm's logic increased automation compliance and task safety, whereas displaying confidence estimates failed to improve performance and reduced compliance on difficult tasks.

Methodology

simulator

Sample size: 72

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• trust calibration
• automation complacency bias
• automation surprise
• automation
• trust in automation foundations

Information type

What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).

• Empirical Findings: self report data