Benchmarks provide common ground for model development: Reply to Logie (2018) and Vandierendonck (2018).

Oberauer, Klaus; Lewandowsky, Stephan; Awh, Edward; Brown, Gordon D. A.; Conway, Andrew R. A.; Cowan, Nelson; Donkin, Chris; Farrell, Simon; Hitch, Graham J.; Hurlstone, Mark J.; Ji, Wei; Morey, Candice C.; Nee, Derek Evan; Schweppe, Judith; Vergauwe, Evie; Ward, Geoff · 2018 · Psychological Bulletin

DOI: 10.1037/bul0000165

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper serves as a response to commentaries by Logie and Vandierendonck regarding the authors’ earlier proposal of benchmarks for evaluating theories and models of short-term and working memory. The authors address two primary criticisms: the perceived narrowness of the benchmark scope and the utility of the proposed rating system for theory development. Regarding scope, the authors defend the exclusion of findings related to working memory’s role in complex cognition (e.g., arithmetic, reasoning, language processing) and neuropsychological case studies. They argue that complex cognitive tasks require assumptions about multiple processes beyond working memory, making them poor tools for adjudicating between competing working memory theories. Conversely, simple, artificial tasks isolate working memory mechanisms more effectively. Neuropsychological case studies were excluded due to difficulties in establishing replicability and generality, though the authors acknowledge their historical importance and hope to include robust findings in future updates. The authors also explain the limited inclusion of visual and spatial materials, noting that subsuming diverse spatial effects under a single "complexity" category would represent an unjustified theoretical commitment. Regarding executive functions, the authors clarify that they included findings on the relationship between working memory and executive functions (such as updating and cognitive control) but avoided the term "executive function" itself due to its ill-defined nature, instead using more precise operational terms. On the issue of utility, the authors defend the A, B, and C rating system, which prioritizes findings based on replicability, generality, and theoretical leverage. They argue that theories must first explain high-priority (A-rated) benchmarks before addressing nuanced (B or C-rated) phenomena, analogous to how astronomical models must explain general planetary motions before predicting specific eclipses. The authors contend that this hierarchy prevents theorists from invoking specialized mechanisms that fail to capture general phenomena. They address concerns that the ratings reflect historical biases or popularity by asserting that the selection process involved consensus among diverse experts and that A-rated findings attract research because they speak to important theoretical questions. Finally, the authors reject the notion that benchmarks constrain research, arguing instead that they provide necessary common ground to prevent the field from becoming overly complex and theoretically stagnant. They emphasize that no current theory jointly explains even the A-rated benchmarks, making these targets essential for advancing rigorous model development.

Key finding

The authors defend the selection and prioritization of specific benchmarks for working memory theories, arguing that this approach provides a necessary common ground for model development despite criticisms regarding its scope.

Methodology

review

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via author_sweep_intake on 2026-05-28.

Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• working memory