Are Vision LLMs Road-Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understanding
archive: archived pipeline: cataloged verified
Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)
Summary
This paper addresses the critical gap in evaluating Vision Large Language Models (VLLMs) for safety-critical autonomous driving applications. While VLLMs have demonstrated strong performance in general visual tasks, their effectiveness in specialized, high-stakes driving scenarios remains largely unexplored. The authors identify three primary limitations in existing benchmarks: the rarity of safety-critical events in current datasets, a lack of temporal-spatial understanding required for dynamic driving contexts, and a narrow evaluation scope that fails to cover comprehensive driving knowledge. To answer whether current VLLMs are "road-ready," the study introduces DVBench, a comprehensive benchmark designed to rigorously assess model capabilities in perception and reasoning within complex, safety-critical driving videos. DVBench is built upon a hierarchical ability taxonomy aligned with established frameworks for assessing highly automated driving systems, such as PEGASUS and NHTSA standards. The taxonomy comprises 25 level-three abilities across perception, reasoning, and causal analysis. The benchmark features 10,000 multiple-choice questions with human-annotated ground-truth answers, curated from videos of crashes and near-crash incidents to ensure realistic and challenging scenarios. To mitigate position-based biases common in large language models, the authors introduce a novel assessment strategy called GroupEval, which analyzes models based on grouped capabilities and incorporates randomized option ordering. This approach ensures fairer and more accurate evaluations of dynamic driving environments. The study evaluates 14 state-of-the-art VLLMs, ranging from 0.5 billion to 72 billion parameters, using the DVBench framework. The results reveal significant performance gaps, with no model achieving accuracy above 40% under the GroupEval strategy. This finding underscores critical limitations in current models' ability to perform high-level driving perception and reasoning in safety-critical contexts. To probe adaptability, the authors fine-tuned selected models using domain-specific data from DVBench. This targeted adaptation resulted in accuracy gains ranging from 5.24 to 10.94 percentage points, with relative improvements of up to 43.59%. These results highlight the necessity of specialized domain knowledge to bridge the gap between general-purpose vision-language models and mission-critical driving applications. The significance of this work lies in establishing DVBench as an essential evaluation framework and research roadmap for developing VLLMs that meet the safety and robustness requirements of real-world autonomous systems. By identifying specific model weaknesses and providing a structured evaluation protocol, the benchmark facilitates the advancement of traffic safety technologies. The authors publicly release the DVBench evaluation toolbox and fine-tuned models to ensure accessibility and reproducibility, encouraging further research into adapting multimodal AI for reliable, safety-critical autonomous driving.
Key finding
No evaluated Vision Large Language Model achieved over 40% accuracy on the safety-critical driving benchmark, but fine-tuning with domain-specific data yielded accuracy gains of up to 10.94 percentage points.
Methodology
dataset
Sample size: 14
Provenance
The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.
| Stage | Outcome | Tool | Model | Prompt | Attempts | Completed |
|---|---|---|---|---|---|---|
| discover | success | — | — | — | 1 | 2026-05-28 |
| archive | success | canonical_url | — | — | 4 | 2026-06-06 |
| extract | success | cached | — | — | 3 | 2026-06-10 |
| clean | success | clean | — | — | 1 | 2026-06-07 |
| chunk | success | chunk | — | — | 1 | 2026-06-07 |
| embed | success | embed | Qwen/Qwen3-Embedding-8B | — | 1 | 2026-06-07 |
| enrich | success | semantic_scholar | — | — | 4 | 2026-07-02 |
| promote | success | — | — | — | 1 | 2026-06-04 |
| summarize | success | llm | qwen3.6-27b-prismaquant | summ-v5 | 2 | 2026-06-10 |
| tag | success | vector_similarity | — | — | 15 | 2026-06-11 |
| verify | success | — | — | — | 2 | 2026-06-10 |
Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.
Topics
Ranked by relevance to this paper. Hover a topic for its definition.
Information type
What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).
- Methodological Resource: dataset resource, tool software
- Theoretical Contribution: computational model