Using Video Analytics to Automatically Annotate Driver Behavior and Context in Naturalistic Driving Data

Sarkar, Abhijit; Abbott, Lynn; Hickman, Jeffrey S.; Papakis, Ioannis; Winkowski, Calvin; Datta, Debanjan; Sonth, Akash; Bhagat, Hirva; Bhat, Shreyas; Jain, Sandesh; Svetovidov, Andrei; Hanowski, Richard; Kaskar, Omkar · 2024 · ROSA P / United States. Department of Transportation. Federal Highway Administration. Office of Research, Development, and Technology

archive: archived pipeline: cataloged verified

Get this paper ↗ (full text — opens at the source; we link to it, we don't host it)

Summary

This research summary report details a project funded by the Federal Highway Administration (FHWA) and conducted by the Virginia Tech Transportation Institute (VTTI) to develop automated video analytics for annotating naturalistic driving data. The study addresses the high cost, time consumption, and potential bias associated with manually annotating large datasets, such as the second Strategic Highway Research Program (SHRP2) Naturalistic Driving Study (NDS). The primary objective was to create a system capable of automatically characterizing driver behavior, classifying the external driving environment, and analyzing interactions between drivers and their surroundings to support transportation safety research and advanced driver assistance systems (ADAS). The researchers employed deep neural network (DNN) models, including convolutional neural networks (CNNs) for image recognition and transformer-based models for sequential data, utilizing transfer learning to adapt pretrained models to driving-specific contexts. The study utilized the SHRP2 NDS dataset, which contains data from over 3,400 drivers and 32 million miles, alongside other specialized datasets for pose estimation and object detection. The methodology focused on four key areas: estimating driver body and head pose to determine gaze direction; classifying secondary behaviors using a Visual Dictionary of Human Action (VDHA); detecting in-vehicle objects and passengers; and analyzing the external environment, including work zones, intersections, and vulnerable road users. The results demonstrated high accuracy across various annotation tasks. For body pose estimation, the HRNet model achieved recall values above 90% for actions such as eating, texting, and interacting with the center stack. Gaze prediction algorithms processed images at 375 frames per second, with temporal models outperforming single-image CNNs in predicting fixation targets. In-vehicle object detection yielded a mean average precision of 56.0% for phones, while passenger detection achieved accuracies between 94.3% and 99.5% depending on the seat position and lighting conditions. External context analysis showed that work zone classification achieved 95% accuracy for work zones and 97% for non-work zones, while intersection detection reached 97% accuracy. The system also successfully estimated traffic density and identified driving environments, though urban and residential classification was less accurate than highway detection. The significance of this work lies in its ability to automate the annotation of 37 different variables in the SHRP2 NDS data dictionary, enabling researchers to analyze driver distractions and environmental interactions at an unprecedented scale. The findings suggest that these computer vision techniques can facilitate the development of robust ADAS capable of detecting distraction and warning drivers. By making the code and models open source, the project provides a foundational toolset for human factors research, allowing for the exploration of safety questions that were previously inaccessible due to the limitations of manual data processing.

Key finding

Deep neural network-based video analytics can automatically and accurately annotate driver behaviors, in-vehicle occupants, and external driving contexts in naturalistic driving data with high precision, facilitating scalable safety research.

Methodology

naturalistic

Sample size: 3400

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via bulk_ingest_rosap on 2026-05-23 (6 acquisition events logged).

Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• distraction detection algorithms
• gaze based attention detection
• naturalistic crash near crash
• eye movements scanning
• visual
• situational awareness

Information type

What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).

• Empirical Findings: observational prevalence
• Methodological Resource: dataset resource, tool software