Variational Autoencoder for End-to-End Control of Autonomous Driving with Novelty Detection and Training De-biasing

Amini, Alexander; Schwarting, Wilko; Rosman, Guy; Araki, Brandon; Karaman, Sertac; Rus, Daniela · 2018 · Crossref

DOI: 10.1109/iros.2018.8594386

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the critical safety and reliability challenges in end-to-end autonomous driving, where deep neural networks (DNNs) often fail without warning when encountering scenarios with insufficient or biased training data. The authors propose a novel Variational Autoencoder (VAE) architecture that integrates end-to-end control with self-supervised learning of latent variables. This approach aims to provide two key capabilities: novelty detection to identify situations where the model lacks confidence, and automated dataset debiasing to accelerate training convergence by addressing class imbalances, such as the underrepresentation of turns compared to straight driving. The method employs a semi-supervised VAE structure comprising an encoder and a decoder. The encoder processes raw RGB images (66x200x3 pixels) to map them into a low-dimensional latent space. Unlike standard VAEs, one latent variable is explicitly supervised to predict the vehicle’s steering curvature, while the remaining variables are learned unsupervised to capture semantic features of the scene. The decoder reconstructs the input image from these latent variables. The total loss function combines three components: a supervised mean squared error for steering control, an L1 reconstruction loss for the image, and a Kullback-Leibler divergence term to regularize the latent space. Uncertainty is estimated by propagating the variance of the latent variables through the decoder to compute pixel-wise uncertainty in the reconstructed image, serving as a metric for novelty detection. The model was evaluated on a dataset collected from a full-scale autonomous Toyota Prius in the Boston metropolitan area, featuring synchronized camera, steering, and inertial measurement unit data. The authors determined that 25 latent variables provided the optimal balance between model fit and reconstruction quality. For novelty detection, the system successfully identified over 97% of novel image frames collected at different times of day and detected 100% of camera sensor malfunctions. For dataset debiasing, the authors utilized the learned latent distributions to resample the training data, prioritizing rare events (e.g., turns) and reducing overrepresented samples (e.g., straight roads). This adaptive resampling strategy resulted in accelerated and sample-efficient training, improving the model's ability to handle diverse driving conditions without manual specification of bias corrections. The significance of this work lies in its ability to transform end-to-end control models from "black boxes" into systems capable of self-assessment and adaptive learning. By leveraging latent variable distributions, the proposed architecture provides a quantitative measure of confidence, allowing the vehicle to detect novel or unsafe scenarios. Furthermore, the automated debiasing mechanism addresses the inherent imbalance in real-world driving data, leading to more robust and efficient training pipelines. This approach enhances the safety and reliability of autonomous systems by ensuring they can gracefully handle unanticipated situations and learn effectively from biased datasets.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-prismaquant on 2026-06-26; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• situational awareness