SMc2f: Robust Scenario Mining for Robotic Autonomy from Coarse to Fine

Chen, Yifei; Greer, Ross · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2601.12010

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the challenge of efficiently and accurately mining rare, safety-critical scenarios from massive autonomous vehicle (AV) driving logs. Validating AV safety requires identifying "long-tail" events, such as unexpected pedestrian crossings or aggressive cut-ins, which are crucial for stress-testing decision-making algorithms but exceedingly rare in real-world data. Existing methods, particularly the baseline RefAV framework, rely on Large Language Models (LLMs) to generate executable code that filters trajectory labels. However, RefAV suffers from several limitations: it ignores direct visual evidence from RGB images, depends heavily on the accuracy of upstream 3D detection systems, and experiences computational inefficiency and code generation errors due to zero-shot LLM prompting. To overcome these issues, the authors propose SMc2f (Robust Scenario Mining for Robotic Autonomy from Coarse to Fine), a multi-stage pipeline that integrates vision-language models, knowledge-base-guided prompting, and contrastive learning to improve both retrieval quality and efficiency. The SMc2f framework operates on the Argoverse 2 dataset, which contains 1,000 driving logs and 10,000 natural language queries. The method employs a coarse-to-fine strategy consisting of three components. First, a CLIP-based multimodal coarse filter directly compares natural language queries with raw RGB video frames using cosine similarity. This stage rapidly identifies temporally relevant segments, pruning the search space before any symbolic reasoning occurs. Second, to address LLM instability, the authors construct a Scenario Mining Knowledge Base containing validated triplets of queries, trajectories, and executable Python scripts. During inference, semantically similar examples are retrieved from this base to provide few-shot prompting, guiding the LLM to generate more robust code with fewer runtime errors. Third, a Dual-Encoder Text–Trajectory Matcher (DETTM) performs fine-grained filtering. DETTM uses contrastive learning to embed text descriptions and spatio-temporal trajectories into a shared space, allowing it to re-rank candidate tracks and reduce false positives by pulling matched pairs together and pushing mismatched ones apart. Experiments on public datasets demonstrate that SMc2f achieves substantial gains in both retrieval accuracy and computational efficiency compared to the RefAV baseline. By leveraging visual cues for initial filtering, the system significantly reduces the volume of data processed by the LLM, lowering inference costs while preserving recall. The few-shot prompting mechanism mitigates the hallucinations and syntactic errors common in zero-shot code generation, leading to more reliable scenario extraction. Furthermore, the contrastive matcher provides precise alignment between language and trajectory data, effectively refining the results produced by the code generation stage. The authors validate their approach using the Argoverse 2 Scenario Mining Challenge benchmark, showing that the integrated pipeline successfully handles complex, compositional queries that require reasoning about causality and spatial relationships. The significance of this work lies in its contribution to scalable and automated safety validation for autonomous systems. By bridging the gap between natural language descriptions and raw sensor data, SMc2f offers a more robust alternative to rigid, rule-based mining systems. The introduction of a knowledge-base-guided approach for LLM code generation provides a practical solution to the brittleness of zero-shot program synthesis in safety-critical applications. Additionally, the coarse-to-fine design establishes a template for efficient multimodal retrieval in large-scale driving datasets, enabling developers to extract actionable insights from petabytes of uncurated data. This method facilitates targeted simulation and regression testing, ultimately supporting the construction of comprehensive safety cases for AV deployment.

Key finding

The proposed SMc2f framework achieves substantial improvements in retrieval quality and efficiency for natural language-based scenario mining by combining coarse visual filtering with robust, few-shot conditioned code generation and fine-grained contrastive matching.

Methodology

simulation_modeling

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via author_sweep_intake on 2026-05-28.

Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• situational awareness