Low-Latency Embedded Driver Monitoring System with a Multi-Task Neural Network
DOI: 10.48550/arXiv.2605.02563
archive: archived pipeline: cataloged verified
Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)
Summary
This paper addresses the critical need for real-time, low-latency Driver Monitoring Systems (DMS) to mitigate traffic accidents caused by human factors such as distraction and fatigue. While camera-based DMS solutions are non-invasive and reliable, their widespread adoption is hindered by the high computational cost and latency of complex computer vision pipelines. The authors propose a lightweight, multi-task neural network (MT-CNN) designed for deployment on resource-constrained edge devices, specifically within the NVIDIA Jetson ecosystem. The system aims to simultaneously predict multiple driver state indicators in a single forward pass, satisfying stringent real-time requirements while minimizing computational overhead. The methodology centers on a novel MT-CNN architecture based on MobileNet-v2, utilizing depthwise separable convolutions and inverted residual blocks to balance efficiency and performance. The model processes cropped face regions to output a 209-element vector containing 98 facial landmarks, eyelid opening levels, eye visibility, mouth opening classification, head orientation angles, and distraction classification (phone use or smoking). Due to the lack of public datasets covering all tasks, the authors trained the model using the Landmark-guided Face Parsing (LaPa) dataset augmented with pseudo-labeled images for distraction detection. The full pipeline includes an SSD-based face detector, a SORT tracking algorithm to reduce detection frequency, and a heuristic-based decision unit that aggregates metrics into a "safeness score" via a Finite State Machine. Experimental results demonstrate that the system achieves real-time performance on both entry-level Jetson Nano and more powerful Xavier NX boards. The MT-CNN variants (Tiny, Small, Large) show trade-offs between accuracy and speed; for instance, the Large model achieved a Normalized Mean Error of 2.163 for landmarks and 98.3% accuracy for eye tasks, with inference latencies of 6.71 ms on Xavier NX using FP16 precision. The face detector, optimized with TensorRT, added minimal latency (1.01 ms on NX). End-to-end latency on the Xavier NX ranged from 16.46 ms to 23.73 ms depending on detection frequency, meeting the requirements for real-time monitoring. The system successfully identifies states ranging from "Safe" to "Danger" based on calibrated thresholds for PERCLOS, mouth opening, and head pose. The significance of this work lies in providing a complete, open-source, end-to-end DMS pipeline capable of running on low-power embedded hardware. By integrating multiple tasks into a single efficient network and optimizing the inference workflow, the authors demonstrate that robust driver monitoring is feasible without high-end computing resources. This approach facilitates the broader adoption of DMS in automotive applications, offering a cost-effective solution for enhancing road safety through continuous, real-time assessment of driver attentiveness and engagement.
Key finding
The proposed multi-task neural network enables real-time driver monitoring on low-power edge devices by simultaneously predicting multiple behavioral indicators with low latency and high accuracy.
Methodology
lab_experiment
Provenance
The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.
| Stage | Outcome | Tool | Model | Prompt | Attempts | Completed |
|---|---|---|---|---|---|---|
| discover | success | — | — | — | 1 | 2026-05-07 |
| archive | success | canonical_url | — | — | 1 | 2026-06-06 |
| extract | success | cached | — | — | 3 | 2026-06-10 |
| clean | success | clean | — | — | 1 | 2026-06-04 |
| chunk | success | chunk | — | — | 1 | 2026-06-04 |
| embed | success | embed | Qwen/Qwen3-Embedding-8B | — | 1 | 2026-06-04 |
| enrich | success | — | — | — | 1 | 2026-05-07 |
| promote | success | — | — | — | 1 | 2026-06-04 |
| summarize | success | llm | qwen3.6-27b-prismaquant | summ-v5 | 2 | 2026-06-10 |
| tag | success | vector_similarity | — | — | 15 | 2026-06-11 |
| verify | success | — | — | — | 2 | 2026-06-10 |
Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.
Topics
Ranked by relevance to this paper. Hover a topic for its definition.
Information type
What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).
- Empirical Findings: physiological data
- Methodological Resource: tool software
- Theoretical Contribution: theory or model