Mitigating undesirable emergent behavior arising between driver and semi-automated vehicle

Melman, Timo; Beckers, Niek; Abbink, David · 2020 · arXiv

archive: archived pipeline: cataloged verified

Get this paper ↗ (full text — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the challenge of undesirable emergent behavior in joint human-robot systems, specifically focusing on the interaction between drivers and semi-automated vehicles (sAVs). The authors argue that such behavior cannot be predicted by analyzing the driver or the vehicle algorithms in isolation. Instead, it arises from the dynamic interaction between the two agents and their environment. Current automation algorithms often optimize reward functions based on data gathered in isolation, which may fail to generalize or account for complex human needs. This misalignment can lead to behavioral adaptations such as disuse (rejecting automation due to annoyance) or misuse (over-reliance leading to deskilling or inappropriate responses), which undermine the safety and comfort benefits of automation. To mitigate these issues, the authors propose a three-pronged approach grounded in human factors knowledge. First, they advocate for including driver behavioral mechanisms directly into the sAV’s algorithms and reward functions. Drawing on the concept of bounded rationality, they suggest using risk-based reward functions that model how drivers naturally adapt to environmental changes, such as road width or oncoming traffic. This approach aims to align the vehicle’s actions with the driver’s underlying preferences and risk tolerance. Second, the authors recommend incorporating model-based approaches that predict and account for interaction-induced driver behavioral adaptations. By understanding how drivers adjust their behavior in response to automation (e.g., speeding up when supported by haptic assistance), designers can create systems that nudge drivers toward safe equilibria rather than exacerbating risky behaviors. Third, the paper emphasizes the necessity of driver-centered interaction design to handle residual misalignments. Since models and reward functions are simplified representations, they cannot prevent all undesirable emergence. The authors advocate for haptic shared control, which makes misalignments tangible and correctable, fostering mutual support and awareness between the driver and the sAV. This design allows for transparent communication of each agent’s intentions and capabilities, enabling mutual learning and adaptation. The significance of this work lies in the concept of "symbiotic driving," which combines these three approaches. The authors cite recent test-track and simulator studies demonstrating that symbiotic driving outperforms designs that rely solely on improved reward functions or interaction designs alone. Furthermore, simulator studies indicate that this integrated approach successfully avoids undesirable emergent behavior. The paper concludes by suggesting that future work should explore symbiotic driving as a means to foster beneficial emergent behavior, not only between the driver and the vehicle but also among multiple road users, thereby enhancing overall safety and efficiency in automated driving scenarios.

Key finding

Mitigating undesirable emergent driver-sAV behavior requires complementing reinforcement-learning algorithm design with human factors knowledge through a three-pronged approach: (a) embed bounded-rationality, risk-based driver behavioral mechanisms inside the sAV's reward function; (b) explicitly model interaction-induced behavioral adaptation (disuse, misuse, deskilling) using homeostasis-style theories; and (c) adopt driver-centered interaction design such as haptic shared control that makes misalignment tangible, correctable, and mutually communicable. The combination ('symbiotic driving') outperforms designs that improve only the reward function or only the interaction, in both test-track and simulator evaluations.

Methodology

theoretical

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via discover_arxiv on 2026-05-04 (4 acquisition events logged).

Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• automation surprise
• automation
• behavioral adaptation risk compensation
• situational awareness
• trust calibration
• driverless ads

Information type

What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).

• Theoretical Contribution: computational model, theory or model, conceptual framework