Gaze Supervision for Mitigating Causal Confusion in Driving Agents
DOI: 10.1109/IV55156.2024.10588498
archive: archived pipeline: cataloged verified
Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)
Summary
This paper addresses the problem of "causal confusion" in imitation learning (IL) agents for autonomous urban driving. IL algorithms often fail to generalize because they infer causality from correlated state-space elements rather than explicit causal structures, leading to policies that perform correct actions for incorrect reasons. This issue is particularly pronounced in complex scenarios, such as when agents misattribute importance to non-causal scene elements like the base of a traffic light or opposing traffic movements. The authors propose mitigating this by leveraging human eye gaze, a naturally occurring signal highly correlated with the causal elements of the state space, as a supervisory signal to guide the agent’s attention. The study employs a CARLA-based VR driving simulator called DReyeVR to collect human driving demonstrations alongside eye gaze data. Seven drivers completed five routes each, resulting in a dataset of approximately 70 minutes of driving data. The gaze data was pre-processed to filter noise and aggregated into attention maps. The authors fine-tuned a pre-trained Learning by Cheating (LBC) model using a novel gaze-based contrastive supervision method. This method utilizes a triplet loss formulation where the anchor is the original input, the negative sample involves blurring gazed-at regions, and the positive sample involves blurring non-gazed regions. This encourages the policy network to rely on visual information in fixated regions for decision-making. The experiments compared the vanilla pre-trained model against versions fine-tuned with control-only supervision and those using both control and gaze supervision. Results demonstrate that gaze-based supervision significantly improves both the alignment of the model’s internal attention with human gaze and overall driving performance. Using the Longest6 benchmark, the model fine-tuned with mixed data (rule-based expert and human demonstrations) and both control and gaze losses achieved a Driving Score of 9.61, outperforming the control-only supervised model (7.81) and the vanilla pre-trained model (7.01). Additionally, the Intersection over Union (IoU) between the model’s saliency maps and human attention maps increased from 0.13 in the pre-trained model to 0.18 in the gaze-supervised model, indicating that the agent’s decision-making process better matched human visual attention. The significance of this work lies in providing a practical, non-intrusive method to improve the robustness and interpretability of IL driving agents. By using eye gaze as a supervisory signal, the approach reduces the need for expensive causal structure learning or expert interventions. The findings suggest that aligning agent attention with human gaze can mitigate causal confusion, leading to safer and more generalizable driving policies. The authors note that while this method is promising, rigorous safety verification of human demonstrations is essential for on-road deployment to ensure only exemplary behaviors are distilled into the policy.
Key finding
Fine-tuning imitation learning driving agents with gaze-based contrastive supervision improves both the alignment of model saliency with human attention and overall driving performance compared to control-only supervision.
Methodology
simulator
Sample size: 7
Provenance
The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via qwen3.6_summarize on 2026-05-28 (2 acquisition events logged).
| Stage | Outcome | Tool | Model | Prompt | Attempts | Completed |
|---|---|---|---|---|---|---|
| discover | success | — | — | — | 1 | 2026-05-07 |
| archive | success | — | — | — | 1 | 2026-05-07 |
| extract | success | cached | — | — | 2 | 2026-06-10 |
| clean | success | — | — | — | 1 | 2026-06-01 |
| chunk | success | — | — | — | 1 | 2026-06-01 |
| embed | success | — | — | — | 1 | 2026-06-02 |
| enrich | success | — | — | — | 1 | 2026-05-07 |
| promote | success | — | — | — | 1 | 2026-05-07 |
| summarize | success | llm | qwen3.6-27b-prismaquant | summ-v5 | 3 | 2026-06-10 |
| tag | success | vector_similarity | — | — | 16 | 2026-06-11 |
| verify | success | — | — | — | 2 | 2026-06-10 |
Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.
Topics
Ranked by relevance to this paper. Hover a topic for its definition.
Information type
What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).
- Methodological Resource: tool software
- Theoretical Contribution: computational model