doScenes: An Autonomous Driving Dataset with Natural Language Instruction for Human Interaction and Vision-Language Navigation
DOI: 10.1109/itsc60802.2025.11423583
archive: archived pipeline: cataloged verified
Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)
Summary
This paper introduces **doScenes**, a novel dataset designed to facilitate research on human-vehicle instruction interactions for autonomous driving. The authors address the critical need for autonomous vehicles (AVs) to integrate natural language instructions into their motion planning, particularly for short-term directives that directly influence vehicle behavior. Existing datasets often rely on simulated data, predefined action sets, or focus on scene-level reasoning and risk assessment rather than actionable commands. doScenes bridges this gap by providing real-world multimodal sensor data annotated with natural language instructions and referentiality tags, enabling the development of context-aware and adaptive planning systems that can respond to human commands in dynamic environments. The dataset is built upon the **nuScenes** dataset, which provides 1,000 manually selected scenes of real-world driving conditions, including multimodal sensor data (cameras, LiDAR, radar) and 3D bounding box annotations. The authors applied a retroactive annotation process to these 12-second clips, using a heuristic called the "taxi test" to determine what instruction a passenger would give to trigger the observed motion. Five independent annotators provided instructions for each scene, allowing for multiple annotations per clip to capture varied phrasing. Each instruction is tagged with **referentiality** labels indicating whether it refers to static objects (e.g., "stop at the blue sign"), dynamic objects (e.g., "follow the white van"), both, or neither. This tagging helps distinguish instructions that require ongoing observation of dynamic agents from those based on static scene features. The dataset contains a diverse range of instructions, with statistics showing 535 non-referential, 214 static referential, 159 dynamic referential, and 93 instructions referencing both static and dynamic objects. The authors note that most scenes have only one or two annotations, though some have more to reflect multiple possible instructions leading to the same outcome. The paper highlights that doScenes differs from prior works like Rank2Tell (which focuses on ranking visual elements) and DRAMA (which focuses on risk captioning) by emphasizing actionable directives tied to specific objects. It also contrasts with simulation-based datasets like DriveMLM and LMDrive, offering real-world data with flexible, open-ended instructions rather than a fixed set of commands. The significance of doScenes lies in its potential to advance **Vision-Language-Action (VLA)** models for autonomous driving. By linking imperative language to motion, the dataset supports research into interpretable, interactive motion planning and Vision-and-Language Navigation (VLN). The authors suggest that models trained on doScenes could learn to generate trajectories based on natural language or assign linguistic descriptors to trajectories, enhancing transparency and safety. However, they acknowledge limitations, such as the retroactive nature of the annotations, which serves as a proxy for true real-time instruction-response pairs, and the 12-second clip duration, which may not capture multi-step planning. The dataset is publicly available to foster further research in safe and effective human-vehicle collaboration.
Key finding
The doScenes dataset provides a novel resource for training autonomous systems by pairing real-world driving scenes with natural language instructions and referentiality tags derived from a heuristic annotation process.
Methodology
dataset
Sample size: 1000
Provenance
The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via author_sweep_intake on 2026-05-28.
| Stage | Outcome | Tool | Model | Prompt | Attempts | Completed |
|---|---|---|---|---|---|---|
| discover | success | author_sweep | — | — | 2 | 2026-05-28 |
| archive | success | canonical_url | — | — | 1 | 2026-06-06 |
| extract | success | cached | — | — | 3 | 2026-06-10 |
| clean | success | clean | — | — | 1 | 2026-06-07 |
| chunk | success | chunk | — | — | 1 | 2026-06-07 |
| embed | success | embed | Qwen/Qwen3-Embedding-8B | — | 1 | 2026-06-07 |
| enrich | success | semantic_scholar | — | — | 4 | 2026-06-15 |
| promote | success | — | — | — | 1 | 2026-06-04 |
| summarize | success | llm | qwen3.6-27b-prismaquant | summ-v5 | 2 | 2026-06-10 |
| tag | success | vector_similarity | — | — | 15 | 2026-06-11 |
| verify | success | — | — | — | 2 | 2026-06-10 |
Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.
Topics
Ranked by relevance to this paper. Hover a topic for its definition.
Information type
What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).
- Methodological Resource: dataset resource, tool software