Language-Driven Active Learning for Diverse Open-Set 3D Object Detection

Greer, Ross; Antoniussen, Bjørk; Møgelmose, Andreas; Trivedi, Mohan M. · 2025 · Unknown

DOI: 10.1109/wacvw65960.2025.00110

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the challenge of detecting minority and novel objects in 3D autonomous driving scenes, where data-driven models often underperform due to class imbalance and the high cost of dataset annotation. The authors propose VisLED (Vision-Language Embedding Diversity Querying), a language-driven active learning framework designed to select diverse and informative samples from an unlabeled pool. By leveraging vision-language representations, VisLED aims to enhance detection capabilities for underrepresented classes without requiring model-specific uncertainty estimates, making it suitable for both closed-set and open-set learning scenarios. The methodology utilizes the CLIP model to generate vision-language embeddings for egocentric driving scene images. VisLED operates in two modes: "Open-World Exploring," which selects data points most novel relative to existing data using hierarchical clustering, and "Closed-World Mining," which uses zero-shot learning to classify images into known categories before clustering to mine novel instances of specific classes. The authors evaluate this approach on the nuScenes dataset using the BEVFusion 3D object detection model. Experiments compare VisLED against random sampling and entropy-based querying across training set sizes increasing in 10% increments, measuring performance via mean Average Precision (mAP) and nuScenes Detection Score (NDS). Results demonstrate that VisLED consistently outperforms random sampling and offers competitive performance compared to entropy-querying, despite being model-agnostic. At 50% of the data pool, VisLED achieved a 1% mAP gain over random sampling. Notably, the Open-World Exploring setting marginally outperformed Closed-World Mining across most data volumes. Significant performance gains were observed for minority classes; for instance, bicycle and motorcycle detection improved by approximately 10% and 5% mAP respectively at early training stages, as VisLED effectively sampled these rare instances before they were exhausted from the pool. Similarly, truck detection saw up to a 20% mAP improvement. The framework also improved detection for difficult classes like trailers and construction vehicles. The significance of this work lies in its ability to reduce annotation costs and improve safety-critical detection for rare objects without relying on model-specific uncertainty metrics. Because VisLED selects samples based on visual novelty rather than model confusion, it is particularly well-suited for open-set learning where novel classes may appear unexpectedly. The authors conclude that this approach facilitates more efficient data curation, allowing a single annotated dataset to support various models and tasks, and suggest future work in multi-task learning and continual learning settings.

Key finding

The VisLED active learning framework consistently outperforms random sampling and achieves competitive results against entropy-based methods for 3D object detection by leveraging vision-language embeddings to select diverse and novel data samples.

Methodology

dataset

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via author_sweep_intake on 2026-05-28.

Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• distraction detection algorithms