Absence of DOA Effect but No Proper Test of the Lumberjack Effect: A Reply to Jamieson and Skraaning (2019)

Wickens, Christopher D.; Onnasch, Linda; Sebok, Angelina; Manzey, Dietrich · 2020 · Human Factors The Journal of the Human Factors and Ergonomics Society

DOI: 10.1177/0018720820901957

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper serves as a rebuttal to Jamieson and Skraaning (2019), who argued that the "lumberjack effect" in human–automation interaction is not applicable to complex, real-world settings. The lumberjack effect posits a trade-off: higher degrees of automation (DOA) improve routine performance and reduce workload when functioning correctly, but cause more severe performance degradation and loss of situation awareness (SA) when the automation fails. Jamieson and Skraaning claimed their process control simulation data contradicted this model. Wickens et al. argue that Jamieson and Skraaning’s study design contained critical flaws that invalidate their conclusion of non-applicability. The authors critique Jamieson and Skraaning’s methodology across five specific dimensions. First, the nature of the automation failure was inappropriate; the study simulated failures in the underlying pressure relief system rather than failures in the automation support system itself, which is required to test the lumberjack hypothesis. Second, the SA measure used (IPAQ) assessed general system knowledge and meta-cognition rather than dynamic awareness of specific parameter values during failures, which is the critical component of SA in this context. Third, the authors argue that Jamieson and Skraaning understated subjective ratings that actually supported the lumberjack effect, biasing their interpretation toward null results. Fourth, the study likely suffered from insufficient statistical power due to a small sample size (eight crews per condition), making it difficult to detect significant differences in failure performance. Finally, the study lacked a measure of routine task performance, which is essential for establishing the comparative trade-off between routine and off-nominal conditions central to the lumberjack model. The analysis reveals that while Jamieson and Skraaning found no objective performance degradation during failures, their subjective data showed trends consistent with the lumberjack effect. Wickens et al. contend that the experimental design did not properly isolate the effects of automation failure on operator awareness and performance. Consequently, the claim that the lumberjack model offers little value to designers of complex systems is unwarranted. The authors conclude that the applicability of the lumberjack effect to complex, real-world environments remains an open question requiring further rigorous research. They urge future studies to address these methodological limitations, particularly regarding the proper testing of automation failures and the use of valid SA measures, before dismissing the model’s relevance.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via unpaywall+openalex on 2026-05-08.

Summary generated by qwen3.6-27b-prismaquant on 2026-06-26; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• automation complacency bias
• automation