Traffic scene awareness for intelligent vehicles using ConvNets and stereo vision

Guindel, Carlos; Martín, David; Armingol, José María · 2019 · Crossref

DOI: 10.1016/j.robot.2018.11.010

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper presents a vision-based perception system for intelligent vehicles that combines deep learning-based object detection with stereo vision to achieve real-time 3D localization and viewpoint estimation of traffic participants. The research addresses the need for robust, cost-effective perception modules in autonomous driving, particularly in complex urban environments where prior map knowledge may be scarce. While active sensors like LiDAR offer high accuracy, their cost and integration challenges often hinder widespread adoption. The authors propose a pipeline that leverages the rich semantic information from video frames and the geometric data from stereo cameras to identify dynamic obstacles, such as vehicles, pedestrians, and cyclists, while estimating their position and orientation relative to the ego-vehicle. The system is implemented on the IVVI 2.0 research platform and consists of two parallel processing branches. The first branch performs object detection and viewpoint estimation using a modified Faster R-CNN architecture. This deep convolutional network extracts features from the left image of a stereo pair to classify objects, refine bounding boxes, and estimate the observation angle (yaw) by treating orientation as a multinomial classification task across discrete bins. The second branch handles 3D localization using stereo matching algorithms, specifically comparing Semi-Global Matching (SGM) and the CNN-based DispNet, to generate dense disparity maps and 3D point clouds. To account for vehicle movement and road unevenness, the system employs an online auto-calibration procedure that estimates the camera’s extrinsic parameters (pitch, roll, and height) by fitting a plane to the ground points in the point cloud using RANSAC. Finally, the branches fuse at the localization stage, where 3D points within detected bounding boxes are transformed into a vehicle-centric footprint frame to determine the precise 2D ground position and absolute yaw of each object. Experimental results demonstrate that the proposed approach meets the strict real-time requirements of onboard automotive applications, executing the complete pipeline in a fraction of a second on commercially available hardware. The system effectively handles partially occluded objects and provides accurate 3D localization and orientation estimates across a variety of traffic situations. The integration of viewpoint estimation into the detection network adds minimal computational overhead while providing critical data for predicting object motion. The study highlights that stereo vision can provide competitive localization results in the near environment compared to more expensive sensor suites, validating the feasibility of using vision-based systems for robust traffic scene awareness in autonomous vehicles.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-prismaquant on 2026-06-26; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• distraction detection algorithms