Looking for a better fit? An Incremental Learning Multimodal Object Referencing Framework adapting to Individual Drivers
archive: archived pipeline: cataloged verified
Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)
Summary
This paper addresses the challenge of multimodal object referencing in vehicles, where drivers identify external objects using gestures like pointing, gaze, and head pose. Traditional machine learning models for this task are rigid, trained once on static data, and fail to adapt to individual driver differences or dynamic driving conditions. This lack of adaptability leads to poor performance and issues like catastrophic forgetting when new data is introduced. To solve this, the authors propose **IcRegress**, a novel incremental learning framework designed for regression problems. IcRegress allows models to continuously adapt to new drivers, varying driver states (such as handedness and experience), and changing sensor availability without forgetting previously learned behaviors. The study was conducted using a medium-fidelity driving simulator equipped with three 55-inch screens, a force-feedback steering wheel, and sensors capturing hand gestures (via a Time-of-Flight camera), head pose, and gaze (via webcam and RT-GENE framework). Fifty-six participants performed a dual-task experiment involving driving on a 49-km simulated road and referencing specific buildings outside the vehicle. The dataset included multimodal inputs—pointing, gaze, head pose, and speech—synchronized at 20 Hz. The authors split the data by participant to ensure external validity, further categorizing drivers by experience (amateur vs. expert) and handedness. They evaluated IcRegress against traditional machine learning baselines and introduced new practical metrics to assess referencing performance in a scalable manner. The results demonstrate that IcRegress significantly outperforms single-instance trained models across various driver traits and conditions. The incremental learning approach successfully adapted to individual differences, such as handedness and driving experience, while maintaining performance on previously learned tasks. An ablation study highlighted the importance of each modality, showing that the fusion of pointing, gaze, and head pose vectors (specifically the horizontal components) was critical for accurate object disambiguation. The framework proved robust even when specific modalities were unavailable, showcasing its flexibility in real-world scenarios where sensor data might be intermittent. The significance of this work lies in its contribution to personalized, adaptive human-vehicle interaction. By enabling continuous lifelong learning, IcRegress offers a more scalable and generalizable solution than existing "one-size-fits-all" approaches. This enhances driver safety and convenience by ensuring the vehicle interface adapts to the user’s unique behavior and current state. The authors provide the framework as an open-source tool to facilitate reproducibility and further research in adaptive multimodal interfaces for autonomous and semi-autonomous vehicles.
Key finding
The proposed incremental learning framework, IcRegress, outperforms static baseline models in multimodal object referencing tasks by successfully adapting to individual driver traits and varying driving conditions without suffering from catastrophic forgetting.
Methodology
simulator
Sample size: 56
Provenance
The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via openalex_abstract on 2026-05-08 (4 acquisition events logged).
| Stage | Outcome | Tool | Model | Prompt | Attempts | Completed |
|---|---|---|---|---|---|---|
| discover | success | — | — | — | 1 | 2026-05-07 |
| archive | success | openalex | — | — | 7 | 2026-06-06 |
| extract | success | cached | — | — | 3 | 2026-06-10 |
| clean | success | clean | — | — | 1 | 2026-06-04 |
| chunk | success | chunk | — | — | 1 | 2026-06-04 |
| embed | success | embed | Qwen/Qwen3-Embedding-8B | — | 1 | 2026-06-04 |
| enrich | success | openalex | — | — | 3 | 2026-05-08 |
| promote | success | — | — | — | 1 | 2026-05-07 |
| summarize | success | llm | qwen3.6-27b-prismaquant | summ-v5 | 2 | 2026-06-10 |
| tag | success | vector_similarity | — | — | 15 | 2026-06-11 |
| verify | success | — | — | — | 2 | 2026-06-10 |
Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.
Topics
Ranked by relevance to this paper. Hover a topic for its definition.
Information type
What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).
- Methodological Resource: tool software
- Theoretical Contribution: computational model