The inconvenient truth of ground truth errors in automotive datasets and DNN-based detection

Chan, Pak Hung; Li, Boda; Baris, Gabriele; Sadiq, Qasim; Donzella, Valentina · 2024 · Crossref

DOI: 10.1017/dce.2024.39

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This study investigates the impact of ground truth annotation errors in automotive datasets on the performance of deep neural network (DNN) object detectors. As assisted and automated driving systems increasingly rely on supervised learning, the quality of labeled data is critical for safety. The authors address the problem that widely used benchmark datasets, such as KITTI and nuScenes, contain significant labeling inaccuracies—including missing objects, incorrect bounding boxes (BBs), and poor fits—which can negatively affect model training and evaluation. The research aims to quantify these errors, propose enforceable annotation criteria, and determine how different labeling standards influence DNN performance across various architectures. The methodology involved a detailed analysis of annotation errors in the KITTI MoSeg dataset, leading to the definition of three distinct re-annotation criteria: C1 (objective rules regarding minimum size, visibility, and occlusion), C2 (annotations based on the best human visual identification), and C3 (removal of clearly erroneous or fully occluded BBs from the original labels). The KITTI MoSeg dataset was manually re-annotated according to each criterion. Three state-of-the-art object detection architectures—Faster R-CNN (two-stage), YOLOv5 (one-stage), and DETR (transformer-based)—were trained on the original labels and each of the three new annotation sets. The models were evaluated using mean Average Precision at 50% Intersection over Union (mAP50) to assess detection accuracy across different training and testing label combinations. The results demonstrate that ground truth quality significantly affects DNN performance, with improvements of up to 9% in mAP50 achieved through better annotations. Removing incorrect bounding boxes (C3) improved evaluation metrics for all networks by reducing false negatives, though this created a bias due to the reduced number of test objects. Training with C2 labels, which included more objects and smaller instances, generally yielded the best performance for Faster R-CNN and YOLOv5 when tested on realistic labels. However, the transformer-based DETR model performed worse when trained on C3 (cleaned) labels compared to the original noisy labels, suggesting that DETR may benefit from the noise or requires larger datasets to generalize effectively without it. The study also highlights that approximately 10% of annotations remain ambiguous or subjective, particularly regarding occlusion and low-contrast scenarios. The significance of this work lies in its demonstration that annotation errors are not merely noise but systematically impact model reliability and benchmarking fairness. The authors conclude that standardized, enforceable annotation criteria are essential for the automotive community to ensure uniform labeling practices. Furthermore, the findings imply that algorithm design and deployment strategies must account for dataset imperfections, as different network architectures exhibit varying robustness to labeling errors. This research provides a framework for improving dataset quality and offers guidelines for future annotation processes to enhance the safety and performance of automated driving systems.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-prismaquant on 2026-06-26; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• dms validation