Natural Language Instructions for Scene-Responsive Human-in-the-Loop Motion Planning in Autonomous Driving using Vision-Language-Action Models
DOI: 10.48550/arxiv.2602.04184
archive: archived pipeline: cataloged verified
Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)
Summary
This paper addresses the challenge of integrating free-form, natural language instructions from passengers into autonomous vehicle (AV) motion planning. While prior instruction-following planners rely on simulation or fixed command vocabularies, limiting real-world generalization, this work investigates how linguistic conditioning influences trajectory generation in real-world scenarios. The authors utilize the doScenes dataset, which links natural language directives to nuScenes ground-truth motion, to adapt OpenEMMA, an open-source multimodal large language model (MLLM)-based end-to-end driving framework. The primary research question is whether natural language conditioning meaningfully steers predicted driving behavior and improves robustness compared to instruction-agnostic baselines. The methodology involves augmenting OpenEMMA, which typically processes front-camera views and ego-state data to output 10-step speed–curvature trajectories, with passenger-style prompts from doScenes. The authors selected LLaVA-1.6-Mistral-7B as the underlying VLM for reproducibility. They evaluated the system on 849 annotated scenes from nuScenes, comparing trajectories generated with and without doScenes instructions. Performance was measured using Average Displacement Error (ADE), a standard metric for geometric similarity between predicted and ground-truth trajectories. The analysis included filtering for actionable instructions and examining the effects of instruction length, clarity, and referentiality (static vs. dynamic elements). The results demonstrate that instruction conditioning substantially improves robustness by preventing extreme baseline failures, yielding a 98.7% reduction in mean ADE across all scenes. When outlier scenes were removed via a 97.5th-percentile filter, instructions still improved ADE by up to 5.1%. The study found that instructions referencing dynamic objects (e.g., "Follow the yellow car") yielded the lowest ADE values, suggesting that temporal and relational context aids the model. Conversely, non-referential or purely static prompts performed less effectively. Additionally, while longer instructions (19+ words) achieved the lowest absolute ADE, typical-length instructions (9–12 words) provided the greatest relative improvement over the baseline. Qualitative analysis revealed that well-phrased instructions corrected critical safety failures, such as preventing unsafe turns or ensuring stops at crosswalks, which the baseline model often missed. The significance of this work lies in establishing the first reproducible baseline for instruction-conditioned planning in autonomous driving using real-world data. It proves that natural language can actively steer AV motion plans, bridging the gap between language understanding and continuous control. The findings highlight that prompt phrasing and referentiality are critical factors in trajectory accuracy. The authors release code and evaluation scripts to facilitate further research, noting limitations such as the "always-act" bias in current models and the need for future work on closed-loop evaluation and more advanced VLMs to handle unsafe or contradictory instructions.
Key finding
Integrating natural language instructions into the OpenEMMA motion planner significantly reduces prediction errors and prevents extreme baseline failures, confirming that passenger directives can actively steer autonomous vehicle trajectories.
Methodology
dataset
Sample size: 849
Provenance
The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via author_sweep_intake on 2026-05-28.
| Stage | Outcome | Tool | Model | Prompt | Attempts | Completed |
|---|---|---|---|---|---|---|
| discover | success | author_sweep | — | — | 2 | 2026-05-28 |
| archive | success | canonical_url | — | — | 1 | 2026-06-04 |
| extract | success | cached | — | — | 3 | 2026-06-10 |
| clean | success | clean | — | — | 1 | 2026-06-04 |
| chunk | success | chunk | — | — | 1 | 2026-06-04 |
| embed | success | embed | Qwen/Qwen3-Embedding-8B | — | 1 | 2026-06-04 |
| enrich | success | — | — | — | 1 | 2026-05-28 |
| promote | success | — | — | — | 1 | 2026-06-04 |
| summarize | success | llm | qwen3.6-27b-prismaquant | summ-v5 | 2 | 2026-06-10 |
| tag | success | vector_similarity | — | — | 15 | 2026-06-11 |
| verify | success | — | — | — | 2 | 2026-06-10 |
Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.
Topics
Ranked by relevance to this paper. Hover a topic for its definition.
Information type
What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).
- Theoretical Contribution: computational model