Hands on Wheel Classification Based on Depth Images and Neural Networks

Schmitz, Jan-Christoph; Tilgner, Stephan; Kalischewski, Kathrin; Wagner, Daniel; Kummert, Anton · 2020 · Crossref

DOI: 10.1051/matecconf/202030806003

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the critical safety requirement for automated vehicles to verify that a driver is ready to intervene by checking if their hands are on the steering wheel. As autonomous driving transitions toward higher levels of automation, ensuring the driver can take control in rare situations is paramount. The authors propose a system using an artificial neural network to classify hand-on-wheel status based on depth images captured by a Time of Flight (ToF) camera mounted in the vehicle’s roof module. This top-down view provides intensity and depth data of the driver’s cabin. The system is specifically designed for embedded automotive hardware, necessitating a lightweight neural network architecture to minimize computational effort. The experimental setup utilizes a dataset of 90,300 images collected from two different cars with varying driver positions, clothing, and camera versions. The data includes both intensity and depth channels, though only the left half of the images is processed to reduce input size. The classification model consists of three convolutional layers with 3x3 kernels, ReLU activations, and max pooling, followed by two fully connected layers with dropout regularization to prevent overfitting. The network is trained using the Adam optimizer with cross-entropy loss and weight decay. Due to a significant class imbalance in the dataset—where only 32.89% of examples show hands on the wheel—the authors evaluate performance using the Area Under the Precision-Recall Curve (PR AUC) rather than simple accuracy. The results demonstrate that the depth image alone is sufficient for high-performance classification, as adding intensity data yields no improvement. The intensity-only input performs significantly worse. Analysis of the class imbalance reveals that adjusting the classification threshold to approximately 0.2 balances the false positive and false negative rates, achieving a total error rate near the minimum of 5.4%. This adjustment is crucial because the default 0.5 threshold favors the majority class (hands off wheel). Furthermore, learning curve analysis indicates that the model has not yet saturated; performance on the test set continues to improve with more training data, suggesting that expanding the dataset, particularly for underrepresented cases, would likely enhance generalization. The study concludes that while the system performs well, remaining errors are primarily false negatives, occurring when hands are near the thigh or at the image edge. These errors are considered safer than false positives, as the system rarely incorrectly assumes the driver is ready to drive. The findings imply that depth-based ToF cameras are a viable, computationally efficient solution for driver monitoring in embedded automotive systems. Future work should focus on expanding the dataset to cover edge cases and potentially differentiating additional classes, such as occluded views, to further improve robustness.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-nvidia on 2026-08-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• distraction detection algorithms