Real-time Driver Monitoring Systems on Edge AI Device

Hariharan, Jyothi; Varior, Rahul Rama; Karunakaran, Sunil · 2023 · arXiv

archive: archived pipeline: cataloged verified

Get this paper ↗ (full text — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the challenge of deploying real-time Driver Monitoring Systems (DMS) on low-cost edge AI devices, motivated by increasing road accidents due to driver inattention and new regulatory mandates such as Euro NCAP standards. While AI-based DMS offers significant safety benefits, deploying deep learning models on edge hardware is difficult due to memory constraints, limited processing capacity, and incompatibilities between model operators and vendor-specific hardware accelerators. The authors present a camera-based DMS system designed to run on the Texas Instruments TDA4VM edge device, focusing on optimizing inference speed by fully offloading computations to the device’s specialized Matrix Multiply Accelerator (MMA). The system architecture comprises an infrared camera for capturing driver footage, which enables low-light visibility and see-through capability for sunglasses, and an AI stack running on the TDA4VM edge device. The perception pipeline utilizes a face detection model followed by a face landmark model to estimate key facial points, head pose, eye closure, gaze, and yawning, thereby inferring the driver’s alertness state. To overcome hardware limitations where certain deep learning operators are unsupported by the MMA, the authors developed a model graph translation library to perform "model surgery." This process involves replacing unsupported operators, such as dequantize and channel-wise padding layers, with functionally equivalent supported operations like Conv2D filters. Additionally, the models were quantized to 8-bit integer precision to leverage the MMA’s optimization for integer computations. Experimental results demonstrate significant performance improvements through these optimization techniques. Model surgery increased the frames per second (FPS) for the Mediapipe face detection model from 121.55 to 163.60 and for the Mediapipe face landmark model from 259.84 to 500.97. Although quantization errors affected some models, the final DMS system utilized the YOLOX-tiny face detection model and the PFLD face landmark model. This configuration achieved a total system throughput of 63 FPS, exceeding the target real-time speed of 30 FPS. Detailed timing analysis revealed that face detection inference took 6.1 ms and landmark inference took 3.2 ms, with the total processing time per frame at 15.8 ms. The study concludes that fully offloading deep learning workloads to hardware accelerators via model surgery and quantization significantly reduces inference latency, enabling high-performance DMS on edge devices. This approach allows for cost-effective, real-time monitoring solutions that meet automotive safety requirements. The authors note that further optimizations, such as addressing quantization errors in bounding box predictions, remain part of their ongoing development roadmap.

Key finding

After model surgery to fit the TDA4VM SDK and offload all DL operators to the matrix accelerator (MMA), the camera-based DMS achieves 63 FPS inference on the edge device, above the authors' real-time target.

Methodology

other

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via discover_arxiv on 2026-05-04 (4 acquisition events logged).

Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• dms validation