Modeling Drivers' Situational Awareness from Eye Gaze for Driving Assistance

Biswas, Abhijat; Gupta, Pranay; Khurana, Shreeya; Held, David; Admoni, Henny · 2024 · Conference on Robot Learning (CoRL), PMLR v270

archive: archived pipeline: cataloged verified

Get this paper ↗ (full text — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the challenge of modeling drivers' situational awareness (SA) to enable intelligent driving assistance systems that avoid alert fatigue. Current systems often alert drivers to all environmental objects, regardless of whether the driver is already aware of them. To prevent unnecessary alerts, the authors propose a real-time, object-level model of driver SA based on eye gaze and scene context. A primary motivation is the lack of suitable training data; traditional SA measurement methods, such as the Situation Awareness Global Assessment Technique (SAGAT), yield sparse and intermittent labels that are unsuitable for machine learning. The authors aim to create a dense, continuous dataset and a predictive model that leverages both global scene context and local gaze-object relationships. To achieve this, the authors developed a novel interactive labeling protocol within a VR driving simulator (DReyeVR). Twenty participants drove scripted routes while pressing directional buttons on a steering wheel controller to indicate when they became aware of specific traffic objects (vehicles, pedestrians, or two-wheelers). This protocol generated a dataset of 80 episodes, comprising approximately 340 minutes of driving data with continuous, dense object-level SA labels, driving actions, and eye gaze history. The authors formulated the SA prediction problem as a semantic segmentation task. The model inputs a binary object mask of the scene and a gaze history map, which represents a 10-second history of the driver’s gaze transformed into 2D pixel coordinates. The gaze map excludes saccades and uses fading dots to encode temporal recency. The model, built on a Feature Pyramid Network with a MobileNetV2 backbone, outputs a segmentation map classifying pixels as aware, unaware, or background. The results demonstrate that the proposed segmentation-based model outperforms common-sense baselines and prior art. Specifically, the model achieved 72.21% accuracy, 0.73 precision, and 0.77 recall on a held-out test set, surpassing a naive gaze-intersection baseline and a handcrafted feature-based SVM baseline. Ablation studies revealed that the "ignore mask" (which prevents penalizing predictions for objects aware before the gaze history window) and saccade filtering were critical for performance. Additionally, the segmentation formulation allows the model to process all objects in a scene simultaneously in a single inference step, resulting in constant runtime complexity regardless of the number of objects, unlike prior methods that scale linearly with object count. The significance of this work lies in providing a publicly available dataset with continuous SA labels and a scalable model for predicting driver awareness. By accurately identifying which objects a driver is unaware of, this approach enables downstream assistance systems to issue alerts only when necessary, thereby reducing alert fatigue and improving safety. The authors conclude that while their current model is memoryless, the framework establishes a foundation for future work integrating temporal modeling and real-world deployment of intelligent driving assistance.

Key finding

A semantic segmentation-based model that jointly processes scene context and gaze history significantly outperforms gaze-intersection and handcrafted-feature baselines in predicting object-level driver situational awareness.

Methodology

simulator

Sample size: 20

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via qwen3.6_summarize on 2026-05-28 (3 acquisition events logged).

Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• situational awareness
• gaze based attention detection
• attention allocation
• eye movements scanning
• looked but failed to see
• anticipation

Information type

What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).

• Methodological Resource: tool software, measurement protocol
• Theoretical Contribution: theory or model