STDA: Spatio-Temporal Dual-Encoder Network Incorporating Driver Attention to Predict Driver Behaviors Under Safety-Critical Scenarios

Xu, Dongyang; Luo, Yiran; Lu, Tianle; Wang, Qingfan; Zhou, Qing; Nie, Bingbing · 2024 · arxiv

DOI: 10.48550/arXiv.2408.01774

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the challenge of accurately predicting driver behavior in safety-critical scenarios, a task often neglected by existing models that perform well only in regular driving conditions. The authors argue that traditional approaches overlook driver attention (DA), a crucial cognitive indicator for hazard perception. To bridge this gap, they propose STDA (Spatio-Temporal Dual-Encoder Network Incorporating Driver Attention), a model designed to mimic human situational awareness by integrating predicted driver attention with visual data to improve both prediction accuracy and interpretability. The STDA architecture comprises four modules: a driver attention prediction module using MobileNet-V2 and Conv-GRU to identify visually prominent regions; a fusion module that combines these attention heatmaps with original first-person image streams; a temporal encoder module to process dynamic scene features; and a behavior prediction module using CNNs and MLPs. The authors developed two fusion variants: STDA-B, which uses pixel-wise image blending, and STDA-C, which employs a cross-attention mechanism. The model was trained on the Personalized Situation Awareness of Drivers (PSAD) dataset, which contains 2,724 safety-critical video clips with corresponding gaze and emergency response data. Due to significant class imbalance in the dataset (braking being the predominant response), the authors utilized Cost-Sensitive Learning for training and evaluated performance using metrics robust to imbalance, such as Geometric Mean (G-mean) and Index of Balanced Accuracy (IBA). Experimental results demonstrate that STDA outperforms mainstream models, including ResNet, ViT, and Swin Transformer variants. Specifically, STDA-B achieved a G-mean of 0.719, an improvement over baseline models. Ablation studies revealed that incorporating the driver attention module increased the G-mean by 4.5% for STDA-B, while adding the temporal encoder further boosted performance. The combination of both modules yielded the highest gains, increasing G-mean by 8.9% and IBA by 18% for STDA-B. Furthermore, the study validated the generalizability of the proposed modules by integrating them into other existing architectures, resulting in consistent performance improvements across diverse models, with some seeing G-mean increases exceeding 40%. The significance of this work lies in its demonstration that integrating driver attention and temporal encoding significantly enhances behavior prediction in hazardous, imbalanced scenarios. STDA-B proved superior to STDA-C, suggesting that direct image blending preserves spatial coherence better than cross-attention for this task. The findings indicate that these modules can serve as effective, generalizable enhancements for various neural networks, offering a more holistic and interpretable approach to autonomous driving safety systems.

Key finding

Adding predicted driver attention as a fused input channel and using a temporal encoder over historical frames improved G-mean from 0.659 to 0.719 on the PSAD safety-critical behavior prediction task, with consistent gains when the attention/temporal modules were transplanted into other mainstream backbone models.

Methodology

modeling

Sample size: PSAD dataset: 2,724 safety-critical scene videos; training data drawn from a single carefully selected subject (subject 101) with response counts of 1,730 brake, 319 right turn, 264 left turn.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via arxiv_pdf on 2026-05-07 (2 acquisition events logged).

Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• distraction detection algorithms
• situational awareness
• anticipation
• telematics crash prediction

Information type

What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).

• Methodological Resource: tool software
• Theoretical Contribution: computational model, theory or model