Exploiting Three-Dimensional Gaze Tracking for Action Recognition During Bimanual Manipulation to Enhance Human–Robot Collaboration
archive: archived pipeline: cataloged verified
Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)
Summary
This study addresses the challenge of recognizing human actions during bimanual manipulation to enhance human–robot collaboration. While traditional approaches rely on two-dimensional egocentric camera videos, this research investigates whether three-dimensional (3D) gaze behavior features can serve as effective inputs for machine learning algorithms to identify human intent and actions. The motivation stems from the need for robots to intuitively understand human goals, particularly in assistive contexts where users may have mobility impairments. The researchers conducted an experiment with 11 subjects performing a bimanual instrumental activity of daily living: preparing a powdered drink. The task involved six subtasks, including removing a pitcher lid, stirring, and pouring. Data were collected using a marker-based motion capture system and a binocular head-mounted eye tracker to reconstruct 3D gaze vectors and their intersections with 3D point clouds of the manipulated objects. The study analyzed gaze fixation duration, saccade size, and constructed 3D gaze saliency maps. Additionally, the authors defined a "gaze object sequence," capturing the identity and temporal sequence of visually regarded objects, to test a simple action recognition algorithm using dynamic time warping. Statistical analyses revealed distinct gaze patterns associated with specific actions. Actions requiring higher visual attention, such as pouring and stirring, exhibited significantly longer gaze fixation durations compared to reaching, picking up, or moving. Saccade sizes also varied significantly, with "move" actions showing larger saccades and "stir" actions showing smaller ones. The 3D gaze saliency maps encoded action-relevant information across the six subtasks. Using the gaze object sequence as a feature, the action recognition algorithm achieved an average accuracy of 96.4%, with precision and recall of 89.5% and 89.2%, respectively. The findings demonstrate that 3D gaze object sequences are promising features for robust action recognition. The high performance suggests that gaze-based methods can effectively infer human intent during complex manipulation tasks. This approach offers advantages over 2D video analysis by providing view-independent insights into gaze behavior. The authors conclude that integrating these gaze features into sophisticated machine learning classifiers could enable real-time action recognition, thereby improving the quality of life for individuals using assistive robots and enhancing collaboration in industrial environments.
Provenance
The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.
| Stage | Outcome | Tool | Model | Prompt | Attempts | Completed |
|---|---|---|---|---|---|---|
| discover | success | DOAJ | — | — | 1 | 2026-06-19 |
| archive | success | unpaywall | — | — | 1 | 2026-06-25 |
| extract | success | cached | — | — | 2 | 2026-06-26 |
| clean | success | clean | — | — | 1 | 2026-06-19 |
| chunk | success | chunk | — | — | 1 | 2026-06-19 |
| embed | success | embed | Qwen/Qwen3-Embedding-8B | — | 1 | 2026-06-19 |
| promote | success | — | — | — | 1 | 2026-06-19 |
| summarize | success | llm | qwen3.6-27b-prismaquant | summ-v5 | 1 | 2026-06-26 |
| tag | success | vector_similarity | — | — | 6 | 2026-06-19 |
| verify | success | — | — | — | 1 | 2026-06-26 |
Summary generated by qwen3.6-27b-prismaquant on 2026-06-26; verification: verified.
Topics
Ranked by relevance to this paper. Hover a topic for its definition.