Gaze Supervision for Mitigating Causal Confusion in Driving Agents
DOI: 10.65109/lhpf5381
archive: archived pipeline: cataloged verified
Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)
Summary
This paper addresses the problem of "causal confusion" in imitation learning (IL) agents for autonomous urban driving. Causal confusion occurs when IL algorithms infer causality from strongly correlated state-space elements rather than the underlying causal structure, leading to policies that perform correctly in training but fail to generalize at test time because they rely on spurious correlations. The authors propose mitigating this issue by leveraging human driver eye gaze as a supervisory signal. Since gaze naturally highlights the causal elements of the visual state space relevant to decision-making, it provides a non-intrusive, continuous signal that can guide the agent to focus on the correct visual features. The methodology involves collecting human driving demonstrations and eye gaze data using the DReyeVR simulator, which integrates VR driving within the CARLA environment. Seven drivers completed five routes each, resulting in a dataset of approximately 70 minutes of driving data. The gaze data was pre-processed to filter noise and aggregated into attention maps. The authors fine-tuned a pre-trained Learning by Cheating (LBC) model, which is known to suffer from causal confusion, using a novel gaze-based contrastive supervision method. This method employs a triplet loss formulation where the anchor is the original input, the negative input has Gaussian blur applied to gazed-at regions, and the positive input has blur applied to non-gazed regions. This encourages the policy to change its driving decisions based on visual information in fixated regions while remaining robust to changes in unimportant areas. Experimental results demonstrate that gaze supervision improves both the interpretability and performance of the IL agents. Using the Longest6 benchmark, the authors evaluated driving performance via the Driving Score (DS) and model saliency alignment via Intersection over Union (IoU) with human attention maps. Fine-tuning with mixed data (rule-based expert and human demonstrations) and both control and gaze supervision achieved a DS of 9.61 and an IoU of 0.18. This outperformed the pre-trained model (DS 7.01, IoU 0.13) and a model fine-tuned with control supervision only (DS 7.81, IoU 0.12). The results indicate that gaze supervision helps the model’s saliency better match human attention and leads to superior driving performance compared to standard IL approaches. The significance of this work lies in providing a practical, low-cost method to improve the robustness and generalization of autonomous driving policies. By using naturally occurring human gaze data, the approach avoids the need for expensive expert interventions or complex causal structure learning. The findings suggest that incorporating gaze-based supervision can effectively mitigate causal confusion, ensuring that driving agents base their decisions on causally relevant environmental features rather than spurious correlations. This has implications for developing safer and more reliable autonomous systems that align more closely with human reasoning processes.
Key finding
Fine-tuning imitation learning driving policies with gaze-based contrastive supervision improves driving performance and aligns model saliency with human visual attention.
Methodology
simulator
Sample size: 7
Provenance
The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.
| Stage | Outcome | Tool | Model | Prompt | Attempts | Completed |
|---|---|---|---|---|---|---|
| discover | success | Crossref | — | — | 1 | 2026-06-05 |
| archive | success | canonical_url | — | — | 1 | 2026-06-06 |
| extract | success | cached | — | — | 3 | 2026-06-10 |
| clean | success | clean | — | — | 1 | 2026-06-07 |
| chunk | success | chunk | — | — | 1 | 2026-06-07 |
| embed | success | embed | Qwen/Qwen3-Embedding-8B | — | 1 | 2026-06-07 |
| promote | success | — | — | — | 1 | 2026-06-05 |
| summarize | success | llm | qwen3.6-27b-prismaquant | summ-v5 | 2 | 2026-06-10 |
| tag | success | vector_similarity | — | — | 15 | 2026-06-11 |
| verify | success | — | — | — | 2 | 2026-06-10 |
Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.
Topics
Ranked by relevance to this paper. Hover a topic for its definition.
- gaze based attention detection
- looked but failed to see
- situational awareness
- distraction detection algorithms
Information type
What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).
- Methodological Resource: tool software
- Theoretical Contribution: computational model