DrivingResearch.

From Scene to Object: Text-Guided Dual-Gaze Prediction

Ke, Zehong; Jiang, Yanbo; Li, Jinhao; Liu, Zhiyuan; Tu, Yiqian; Meng, Qingwen; Huang, Heye; Wang, Jianqiang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2604.20191

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the critical limitation of existing driver attention prediction models, which rely on scene-level global gaze heatmaps that lack fine-grained, object-level semantic grounding. This data deficiency causes a "text-vision decoupling" problem in Vision-Language Models (VLMs), where textual reasoning fails to align with spatial predictions, leading to visual-bias hallucinations. To resolve this, the authors propose a paradigm shift from global estimation to text-guided, object-level attention mapping, introducing both a new dataset construction pipeline and a novel model architecture. The methodology consists of two main components. First, the authors construct the G-W3DA dataset, an object-level driver attention dataset. They develop an automated annotation pipeline that integrates a multimodal large language model (Qwen3.5-Plus) with the Segment Anything Model 3 (SAM3). This pipeline uses a dual-vision prompting strategy to map grayscale attention heatmaps to semantic descriptions, followed by cascaded zero-shot segmentation to generate object masks. A rigorous cross-validation mechanism filters these masks based on mean attention intensity to eliminate hallucinations, decoupling macroscopic heatmaps into precise, semantically grounded object-level masks. Second, the authors propose the DualGaze-VLM architecture, a dual-branch network that predicts both macroscopic global gaze and microscopic object-level attention. The model extracts semantic queries from reasoning text and uses a Condition-Aware SE-Gate to dynamically modulate visual features, ensuring intent-driven spatial anchoring. Experimental results on the W3DA benchmark demonstrate that DualGaze-VLM consistently outperforms existing state-of-the-art models. The framework achieves a 17.8% improvement in Similarity (SIM) metrics under safety-critical scenarios, highlighting superior spatial alignment. Furthermore, human evaluation tests reveal that 88.22% of evaluators perceived the generated attention heatmaps as authentic, confirming the model’s ability to produce rational cognitive priors. Ablation studies further validate that object-level supervision significantly enhances prediction precision and robustness compared to traditional scene-level approaches. The significance of this work lies in its establishment of a complete paradigm for interpretable driver attention prediction, bridging high-level semantic reasoning with low-level spatial perception. By providing physically grounded, object-level supervision, the study overcomes the modality gap that has hindered VLM integration in autonomous driving. The proposed G-W3DA dataset and DualGaze-VLM architecture offer a robust foundation for future research in cognitive modeling, enabling autonomous systems to better mimic human-like "Observation-Reasoning-Attention" loops and improve safety through precise, text-grounded visual focus.

Key finding

The proposed DualGaze-VLM framework achieves state-of-the-art performance in driver attention prediction by leveraging object-level supervision, resulting in a 17.8% improvement in similarity metrics for safety-critical scenarios and high authenticity ratings from human evaluators.

Methodology

simulation_modeling

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via author_sweep_intake on 2026-05-28.

StageOutcomeToolModelPromptAttemptsCompleted
discover success author_sweep — — 2 2026-05-28
archive success canonical_url — — 1 2026-06-04
extract success cached — — 3 2026-06-10
clean success clean — — 1 2026-06-04
chunk success chunk — — 1 2026-06-04
embed success embed Qwen/Qwen3-Embedding-8B — 1 2026-06-04
enrich success — — — 1 2026-05-28
promote success — — — 1 2026-06-04
summarize success llm qwen3.6-27b-prismaquant summ-v5 2 2026-06-10
tag success vector_similarity — — 15 2026-06-11
verify success — — — 2 2026-06-10

Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

Information type

What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).