Mitigating undesirable emergent behavior arising between driver and semi-automated vehicle
URL: http://arxiv.org/abs/2006.16572v3
archive: archived pipeline: cataloged verified
Abstract
Emergent behavior arising in a joint human-robot system cannot be fully predicted based on an understanding of the individual agents. Typically, robot behavior is governed by algorithms that optimize a reward function that should quantitatively capture the joint system's goal. Although reward functions can be updated to better match human needs, this is no guarantee that no misalignment with the complex and variable human needs will occur. Algorithms may learn undesirable behavior when interacting with the human and the intrinsically unpredictable human-inhabited world, thereby producing further misalignment with human users or bystanders. As a result, humans might behave differently than anticipated, causing robots to learn differently and undesirable behavior to emerge. With this short paper, we state that to design for Human-Robot Interaction that mitigates such undesirable emergent behavior, we need to complement advancements in human-robot interaction algorithms with human factors knowledge and expertise. More specifically, we advocate a three-pronged approach that we illustrate using a particularly challenging example of safety-critical human-robot interaction: a driver interacting with a semi-automated vehicle. Undesirable emergent behavior should be mitigated by a combination of 1) including driver behavioral mechanisms in the vehicle's algorithms and reward functions, 2) model-based approaches that account for interaction-induced driver behavioral adaptations and 3) driver-centered interaction design that promotes driver engagement with the semi-automated vehicle, and the transparent communication of each agent's actions that allows mutual support and adaptation. We provide examples from recent empirical work in our group, in the hope this proves to be fruitful for discussing emergent human-robot interaction.
Summary
Position/short paper (Melman, Beckers, Abbink; TU Delft + Renault; arXiv:2006.16572, 2020) arguing that undesirable emergent behaviour in driver–semi-automated-vehicle systems cannot be prevented by tuning vehicle reward functions alone and proposing a three-pronged design approach: (1) include human-factors models of driver behavioural mechanisms inside the sAV's algorithms and reward functions, (2) use model-based methods that account for interaction-induced driver behavioural adaptations (disuse from misalignment/annoyance, misuse from over-reliance), and (3) adopt a driver-centred interaction design that promotes engagement and transparent mutual communication of intent. The paper draws on examples from the Delft Haptics Lab on shared-control steering and lane-keeping to illustrate how disuse and misuse emerge dynamically and can be mitigated.
Key finding
Mitigating undesirable emergent driver–sAV behaviour requires complementing reinforcement-learning algorithm design with human-factors knowledge: (a) embed driver behavioural models in the sAV reward function, (b) explicitly model interaction-induced behavioural adaptation (behavioural adaptation, deskilling), and (c) design transparent, driver-centred interaction. RL alone cannot solve human-robot misalignment in safety-critical settings.
Methodology
Conceptual/position short paper. No human-subject experiment. Synthesizes prior work from the Delft Haptics Lab on driver–sAV shared control and behavioural adaptation; proposes a three-pronged cybernetic interaction-based design framework (Fig. 1) for mitigating emergent misalignment.
Quality score: 5 / 5