It's all about you: Personalized in-Vehicle Gesture Recognition with a Time-of-Flight Camera

Gomaa, Amr; Reyes, Guillermo; Feld, Michael · 2023 · ACM AutomotiveUI 2023

DOI: 10.1145/3580585.3607153

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the challenge of recognizing dynamic hand gestures in vehicle environments, where standard RGB cameras struggle with varying lighting and high-speed motion, and existing datasets are scarce. The authors propose a personalized, model-adaptation approach using a Time-of-Flight (ToF) camera and a Convolutional Long-Short Term Memory (CNNLSTM) neural network. The goal is to improve recognition accuracy and reduce data requirements by customizing the model for individual users, thereby enhancing safety and user experience in automotive Human-Machine Interaction (HMI). The study collected a new dataset from 83 participants performing six simple gestures (e.g., pointing, swiping, rotating) inside a real vehicle equipped with a ceiling-mounted ToF camera. Participants performed gestures while simulating driving conditions to capture natural variability. The data was preprocessed by standardizing frame rates to 12 fps, truncating sequences to 70 frames, applying Gaussian blur, and using background subtraction for hand segmentation. The CNNLSTM architecture extracts visual features via convolutional layers and temporal dependencies via LSTM layers. The dataset was split into a Universal Background Model (UBM) group for initial training and an Adaptation Set (AS) group for personalization experiments. Results demonstrate that a baseline UBM achieved only 66.9% accuracy, struggling with directional distinctions in swipe and rotate gestures. Single-step adaptation to specific users improved accuracy to 77.8%. Incorporating data augmentation (random translations) further boosted the UBM to 79.9% and the adapted models to 86.4%. The study found that personalization requires minimal data; fine-tuning with as few as two additional gestures per class raised accuracy to 87.8%, while 14–20 gestures per class peaked at approximately 90.5%. Incremental learning experiments confirmed that models could be effectively updated with small batches of user-specific data without requiring full retraining. The significance of this work lies in demonstrating the feasibility of using ToF cameras for in-vehicle gesture recognition and establishing that personalized, incremental learning strategies can achieve high accuracy with limited data. By moving away from a "one-model-fits-all" approach, the method accounts for individual user variations, offering a scalable solution for automotive HMI systems. The findings provide reproducible preprocessing guidelines for ToF data and validate the effectiveness of combining transfer learning, data augmentation, and incremental adaptation for user-centered AI applications.

Key finding

Personalized model adaptation combined with data augmentation increased in-vehicle gesture recognition accuracy from 66.9% to over 90%.

Methodology

field_study

Sample size: 83

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via openalex_abstract on 2026-05-08 (3 acquisition events logged).

Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• distraction detection algorithms