Vision and Language: Novel Representations and Artificial intelligence for Driving Scene Safety Assessment and Autonomous Vehicle Planning
DOI: 10.48550/arxiv.2602.07680
archive: archived pipeline: cataloged verified
Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)
Summary
This paper investigates the integration of vision-language models (VLMs) into autonomous driving systems to enhance safety assessment and decision-making in open-world environments. Motivated by the limitations of traditional perception pipelines, which struggle with ambiguity, novelty, and semantic context, the authors explore how VLMs can provide semantic reasoning capabilities. The study addresses the engineering challenge of integrating these abstract representations into safety-critical perception, prediction, and planning pipelines without introducing misalignment or ambiguity. The authors evaluate three complementary system-level use cases. First, they develop a lightweight, category-agnostic hazard screening approach using CLIP-based image-text similarity. This method generates a low-latency semantic hazard signal by comparing camera frames against natural language prompts (e.g., "hazard on the road") and negative controls. They evaluate this on the COOOL benchmark, which contains unexpected road hazards, supplemented by nominal driving footage to balance the dataset. Performance is measured using temporal intersection-over-union (tIoU) metrics. Second, they examine the integration of scene-level vision-language embeddings into a transformer-based trajectory planning framework (adapted from Motion Transformer) using the Waymo End-to-End Driving Dataset. This experiment tests whether global semantic representations improve geometry-aware motion planning. Third, they investigate natural language as an explicit behavioral constraint using the doScenes dataset and the OpenEMMA planning framework. Here, passenger-style instructions grounded in visual scene elements are injected into the planning prompt to guide short-horizon vehicle behavior. The results reveal distinct outcomes for each integration strategy. The CLIP-based hazard screening effectively detects diverse hazards, with "low visibility" and "animal" prompts achieving the highest Global tIoU scores (0.765 and 0.657, respectively). While the general "hazard" prompt was susceptible to false alarms, a dual-hazard gating strategy improved negative tIoU, reducing false positives. Conversely, naïvely conditioning trajectory planners on global vision-language embeddings did not improve trajectory accuracy, highlighting a misalignment between abstract semantic features and precise geometric planning. However, using natural language as an explicit behavioral constraint successfully suppressed rare but severe planning failures and improved safety-aligned behavior in ambiguous scenarios. The study concludes that realizing the safety benefits of vision-language representations is fundamentally an engineering problem requiring careful system design rather than direct feature injection. VLMs are most effective when used to express semantic risk, intent, and behavioral constraints, rather than as undifferentiated inputs for low-level control. The findings suggest that safe autonomous vehicles require representations that bridge human-interpretable semantics and machine-executable planning, necessitating structured grounding and task-informed extraction of semantic content.
Key finding
Vision-language representations improve autonomous driving safety when used to express semantic risk or behavioral constraints, but naive integration of global embeddings into trajectory planners does not enhance performance.
Methodology
simulation_modeling
Provenance
The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via author_sweep_intake on 2026-05-28 (2 acquisition events logged).
| Stage | Outcome | Tool | Model | Prompt | Attempts | Completed |
|---|---|---|---|---|---|---|
| discover | success | author_sweep | — | — | 3 | 2026-05-28 |
| archive | success | canonical_url | — | — | 1 | 2026-06-04 |
| extract | success | cached | — | — | 3 | 2026-06-10 |
| clean | success | clean | — | — | 1 | 2026-06-04 |
| chunk | success | chunk | — | — | 1 | 2026-06-04 |
| embed | success | embed | Qwen/Qwen3-Embedding-8B | — | 1 | 2026-06-04 |
| enrich | success | — | — | — | 1 | 2026-05-28 |
| promote | success | — | — | — | 1 | 2026-06-04 |
| summarize | success | llm | qwen3.6-27b-prismaquant | summ-v5 | 2 | 2026-06-10 |
| tag | success | vector_similarity | — | — | 15 | 2026-06-11 |
| verify | success | — | — | — | 2 | 2026-06-10 |
Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.
Topics
Ranked by relevance to this paper. Hover a topic for its definition.
Information type
What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).
- Theoretical Contribution: computational model