Cross or Nah? LLMs Get in the Mindset of a Pedestrian in front of Automated Car with an eHMI

Alam, Md Shadab; Bazilinskyy, Pavlo · 2025 · OpenAlex-citations

DOI: 10.1145/3744335.3758477

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This study investigates whether Large Language Models (LLMs) and Vision-Language Models (VLMs) can effectively simulate pedestrian decision-making regarding External Human-Machine Interfaces (eHMIs) on automated vehicles. The research is motivated by the high cost and inconsistency of crowdsourced human evaluations for eHMI designs. By determining if LLMs can reliably predict human willingness to cross streets in response to vehicle messages, the authors aim to establish a scalable, cost-effective prescreening tool for interface development. The researchers evaluated 13 distinct VLM architectures, including ChatGPT-4o, Gemma3:27B, and various LLaVA variants, using a dataset of 227 images of automated vehicles displaying textual eHMIs. These images were previously rated by 1,438 human participants who indicated their confidence in crossing on a scale of 0 to 100. The models were tested under two conditions: a "no memory" condition where each image was assessed independently, and a "memory-enabled" condition where the model retained a history of up to six prior prompt-response pairs. Each condition underwent 15 independent trials. Model outputs were extracted and compared against human benchmarks using Spearman correlation coefficients to measure alignment. The results revealed significant disparities in model performance. In the no-memory condition, Gemma3:27B achieved the highest correlation with human judgments ($r = 0.85$), followed closely by ChatGPT-4o ($r = 0.80$). Conversely, many smaller or less specialized models, such as LLaVA-LLaMA-3 and Moondream, showed weak or negative correlations. When conversational memory was introduced, performance dynamics shifted: ChatGPT-4o maintained strong alignment ($r = 0.81$), while Gemma3:27B’s correlation dropped sharply to $r = 0.23$. Most other models exhibited reduced alignment with human data when memory was enabled. Additionally, DeepSeek-VL2-Tiny demonstrated poor granularity, outputting only discrete confidence levels (0, 75, or 90) and frequently misinterpreting safety warnings, such as assigning high confidence to messages indicating the vehicle was accelerating. The study concludes that while high-capacity models like ChatGPT-4o and Gemma3:27B can approximate human pedestrian judgments, their reliability is highly dependent on architectural sophistication and context handling. The finding that conversational memory generally degraded performance suggests that most VLMs struggle to reset their context between independent scenarios, unlike human pedestrians. Consequently, LLM-based personas show promise for rapid eHMI prescreening, particularly when using advanced models without conversational history, but they are not yet a universal replacement for human evaluation due to inconsistent performance across different architectures.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-prismaquant on 2026-06-26; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• ehmi external hmi
• vru facing ehmi
• generative ai voice assistants