Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning
DOI: 10.1109/wacv57701.2024.00734
archive: archived pipeline: cataloged verified
Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)
Summary
This paper introduces Rank2Tell, a novel multi-modal ego-centric dataset designed to address the lack of transparency and interpretability in autonomous vehicle (AV) and advanced driver assistance systems (ADAS). The authors argue that public trust in these systems depends on their ability to identify critical traffic agents and provide human-interpretable reasoning for their importance. Existing datasets lack comprehensive annotations that combine object importance ranking with natural language explanations. To bridge this gap, the paper presents a dataset and a benchmark model for joint importance ranking and captioning, aiming to enhance visual scene understanding in complex urban traffic scenarios. The dataset was collected using an instrumented vehicle equipped with three video cameras, a Velodyne LiDAR sensor, and high-precision GPS, along with Vehicle Controller Area Network (CAN) data. It comprises 116 video clips, approximately 20 seconds each, captured at diverse urban intersections. Annotation involved five annotators with varying driving experience to account for subjective differences in importance perception. Annotators viewed stitched images overlaid with ego-vehicle speed and intention (straight, left, right) but were restricted to 40 historical frames to simulate realistic driving conditions without future knowledge. The annotation process included identifying important agents, localizing them with 2D bounding boxes, ranking their importance on a three-level scale (Low, Medium, High), and generating free-form natural language captions explaining the reasoning. The schema also captured detailed semantic, spatial, temporal, and relational attributes, such as agent type, visual attributes, actions, location, and motion direction. The paper establishes Rank2Tell as the first dataset to provide dense annotations for both importance ranking and natural language explanation for multiple objects in traffic scenes. Statistical analysis reveals diverse importance levels across different agent types and locations relative to the ego vehicle’s intention. The authors introduce a joint model that utilizes multi-modal 2D and 3D features to simultaneously predict importance levels and generate captions for important agents. This model serves as a baseline benchmark for the dataset. The dataset’s unique features, including object tracking and rich attribute annotations, enable the construction of informative scene graphs and support tasks such as situational awareness enhancement and the development of interpretable models for safety-critical applications. The significance of this work lies in its contribution to the interpretability of autonomous systems. By providing a resource that links visual scene understanding with human-readable reasoning, Rank2Tell facilitates the development of systems that can explain their decision-making processes to users. This capability is crucial for improving driver situational awareness, conveying important decisions, and warning passengers of potential hazards. The dataset’s comprehensive annotations and multi-modal nature offer a valuable tool for researchers working on visual scene understanding, potentially leading to more trustworthy and transparent autonomous driving technologies.
Key finding
The Rank2Tell dataset provides the first comprehensive multimodal resource for joint importance ranking and natural language explanation of traffic agents, enabling benchmarked models to improve the interpretability of autonomous driving systems.
Methodology
dataset
Sample size: 116
Provenance
The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via openalex_abstract on 2026-05-08.
| Stage | Outcome | Tool | Model | Prompt | Attempts | Completed |
|---|---|---|---|---|---|---|
| discover | success | — | — | — | 1 | 2026-05-07 |
| archive | success | canonical_url | — | — | 7 | 2026-06-06 |
| extract | success | cached | — | — | 3 | 2026-06-10 |
| clean | success | clean | — | — | 1 | 2026-06-04 |
| chunk | success | chunk | — | — | 1 | 2026-06-04 |
| embed | success | embed | Qwen/Qwen3-Embedding-8B | — | 1 | 2026-06-04 |
| enrich | success | openalex | — | — | 2 | 2026-05-08 |
| promote | success | — | — | — | 1 | 2026-05-07 |
| summarize | success | llm | qwen3.6-27b-prismaquant | summ-v5 | 2 | 2026-06-10 |
| tag | success | vector_similarity | — | — | 15 | 2026-06-11 |
| verify | success | — | — | — | 2 | 2026-06-10 |
Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.
Topics
Ranked by relevance to this paper. Hover a topic for its definition.
Information type
What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).
- Methodological Resource: dataset resource, tool software