Safety with Agency: Human-Centered Safety Filter with Application to AI-Assisted Motorsports
DOI: 10.15607/rss.2025.xxi.093
archive: archived pipeline: cataloged verified
Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)
Summary
This paper addresses the challenge of integrating AI safety filters into shared autonomy systems without compromising human agency, comfort, or trust. The authors identify a critical limitation in conventional safety filters, particularly "last-resort" strategies that intervene abruptly at the last possible moment to prevent failure. Such discontinuous interventions can cause "automation surprise," eroding user confidence and disrupting the operator’s sense of control, which is detrimental in high-stakes, performance-oriented tasks like motorsports. To solve this, the authors propose a Human-Centered Safety Filter (HCSF) designed to enforce safety through minimal, smooth modifications to human inputs, thereby preserving the operator’s strategic edge and agency. The method introduces a novel, fully model-free Control Barrier Function (CBF) safety constraint, termed Q-CBF. The approach relies on a neural safety value function ($Q$) learned via black-box interactions using model-free reinforcement learning, specifically addressing the "curse of dimensionality" that hinders traditional Hamilton-Jacobi reachability analysis. Unlike traditional CBFs that require explicit knowledge of system dynamics for both synthesis and runtime enforcement, the Q-CBF constraint operates without any dynamical model, making it applicable to complex, black-box systems. At runtime, the HCSF solves an optimization problem to find the safest action that minimally deviates from the human’s intended input, ensuring the system remains within the maximal safe set. The authors validate this approach in *Assetto Corsa*, a high-fidelity racing simulator with black-box dynamics, using a 133-dimensional observation space and a multi-phase training pipeline to expose the agent to dangerous states. Experimental validation involved a comprehensive in-person user study with 83 participants of diverse driving skill levels. The study compared the HCSF against two baselines: unassisted driving and a conventional Last-Resort Safety Filter (LRSF). The results demonstrated that the HCSF significantly improved safety metrics and user satisfaction compared to having no assistance, without compromising human agency or comfort. Crucially, when compared to the conventional LRSF, the HCSF yielded significant gains in perceived human agency, comfort, and overall satisfaction while maintaining robust safety performance. The smooth, minimal interventions of the HCSF, supported by transparent visual cues indicating the magnitude and direction of AI corrections, effectively mitigated the abruptness and confusion associated with traditional safety overrides. The significance of this work lies in its demonstration that principled safety guarantees can be achieved in high-dimensional, black-box environments without requiring explicit system models or sacrificing user experience. By bridging the gap between rigorous safety filtering and human-centered design, the HCSF offers a scalable solution for shared autonomy applications where maintaining operator trust and agency is as critical as preventing failure. This approach provides a framework for deploying AI assistants in safety-critical domains, such as autonomous driving and robotics, ensuring that safety interventions enhance rather than hinder human performance.
Key finding
The human-centered safety filter improves safety and user satisfaction without compromising human agency or comfort compared to no assistance, and enhances agency, comfort, and satisfaction relative to conventional safety filters.
Methodology
simulator
Sample size: 83
Provenance
The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via author_sweep_intake on 2026-05-28.
| Stage | Outcome | Tool | Model | Prompt | Attempts | Completed |
|---|---|---|---|---|---|---|
| discover | success | author_sweep | — | — | 2 | 2026-05-28 |
| archive | success | canonical_url | — | — | 1 | 2026-06-04 |
| extract | success | cached | — | — | 3 | 2026-06-10 |
| clean | success | clean | — | — | 1 | 2026-06-04 |
| chunk | success | chunk | — | — | 1 | 2026-06-04 |
| embed | success | embed | Qwen/Qwen3-Embedding-8B | — | 1 | 2026-06-04 |
| enrich | success | — | — | — | 1 | 2026-05-28 |
| promote | success | — | — | — | 1 | 2026-06-04 |
| summarize | success | llm | qwen3.6-27b-prismaquant | summ-v5 | 2 | 2026-06-10 |
| tag | success | vector_similarity | — | — | 15 | 2026-06-11 |
| verify | success | — | — | — | 2 | 2026-06-10 |
Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.
Topics
Ranked by relevance to this paper. Hover a topic for its definition.
Information type
What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).
- Theoretical Contribution: computational model