Studying Person-Specific Pointing and Gaze Behavior for Multimodal Referencing of Outside Objects from a Moving Vehicle

Gomaa, Amr; Reyes, Guillermo; Alles, Alexandra; Rupp, Lydia Helene; Feld, Michael S. · 2020 · arXiv

DOI: 10.1145/3382507.3418817

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This study investigates person-specific pointing and gaze behaviors for multimodal referencing of objects outside a moving vehicle. While hand gestures and eye gaze are established interaction modalities in automotive contexts, existing methods typically treat them separately or focus on static, in-vehicle scenarios. This research addresses the gap in understanding how these modalities interact dynamically during driving, a safety-critical primary task, and explores individual differences to support user-adaptive systems. The researchers conducted a within-subject experiment using a medium-fidelity driving simulator with 39 participants. Participants drove for 40 minutes while performing a secondary task: referencing specific buildings (Points of Interest) outside the vehicle using right-hand pointing and eye gaze. The experimental design varied environmental density (dense vs. non-dense surroundings), target distance (near vs. far), and driving mode (manual vs. autonomous). Data from hand-tracking cameras and eye-tracking glasses were synchronized and transformed into a common 1D cylindrical coordinate system to calculate horizontal angles relative to the vehicle. Performance metrics included pointing and gaze accuracy, detection time, pointing duration, and gaze frequency, categorized into pre-pointing, during-pointing, and post-pointing phases. Statistical analysis revealed significant effects of environmental density and target orientation on performance. Participants achieved higher pointing and gaze accuracy and faster reaction times in non-dense environments compared to dense ones. Targets on the right side of the road were referenced more quickly and with higher gaze accuracy than those on the left, likely due to the right-handed pointing constraint and higher exposure frequency. Autonomous driving significantly improved gaze accuracy compared to manual driving, though pointing accuracy remained unchanged. Crucially, gaze accuracy was consistently higher than pointing accuracy. Intraclass correlation analysis indicated that approximately 20% of the variance in timing-related behaviors stemmed from individual differences, confirming distinct personal referencing styles. Clustering analysis further identified distinct user groups based on accuracy profiles, such as those with high pointing but low gaze accuracy. The findings underscore the necessity of personalized, multimodal fusion approaches for automotive interaction. The data suggests that gaze should be tracked primarily in short windows before and after pointing, as users rarely maintain gaze on targets during the gesture. The significant individual variability supports the development of adaptive systems that adjust tracking thresholds and modality reliance based on user-specific behavioral patterns. This work provides a foundational dataset and behavioral model for designing robust, safety-conscious interfaces for referencing external landmarks in dynamic driving environments.

Key finding

Pointing and gaze accuracy differ significantly by object side (better for right-side PoIs), distractor density, distance, and driving mode (autonomous vs manual); gaze is significantly more accurate than pointing, and individual differences are large enough to motivate a person-specific (clustering / modality-switching) fusion strategy rather than a global model.

Methodology

simulator

Sample size: 73 recruited; 39 retained after exclusions for technical failure (30), motion sickness (2), or improper task execution (2); plus 17-participant online pre-study for PoI salience.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via discover_arxiv on 2026-05-07 (3 acquisition events logged).

Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• gaze based attention detection
• temporal
• attention allocation
• eye movements scanning
• visual
• visual manual

Information type

What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).

• Empirical Findings: behavioral performance data
• Methodological Resource: tool software, measurement protocol