Driving from Vision through Differentiable Optimal Control
DOI: 10.1109/iros58592.2024.10802306
archive: archived pipeline: cataloged verified
Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)
Summary
This paper introduces DriViDOC (Driving from Vision through Differentiable Optimal Control), a framework designed to learn personalized, human-like autonomous driving controllers from visual demonstrations. The research addresses the need for autonomous vehicles that mimic specific human driving styles to enhance passenger comfort and adoption, while maintaining strict safety guarantees. Traditional imitation learning methods often lack safety assurances or interpretability, whereas standard nonlinear model predictive control (NMPC) requires manual parameter tuning and struggles to adapt to diverse human behaviors. DriViDOC bridges this gap by combining a deep convolutional neural network (CNN) with a differentiable NMPC, enabling end-to-end learning from raw camera images to control actions. The methodology leverages the differentiability of parametric NMPC to allow gradient-based optimization through the control layer. A CNN processes a sequence of front-camera frames to infer latent visual features, which are then mapped to dynamic NMPC parameters via fully connected layers. These parameters adjust the objective function weights and offsets, effectively tailoring the control behavior (e.g., trade-offs between speed and acceleration) to the observed driving style. The system is trained using behavioral cloning on an offline dataset collected from 11 human drivers on a high-fidelity hexapod driving simulator. The dataset comprises over 113,000 samples of images, vehicle states, and control actions across a track with varying curvatures. Training involves pretraining the CNN on all drivers, followed by fine-tuning the combined CNN-NMPC architecture for individual drivers. Experimental results demonstrate that DriViDOC successfully imitates distinct human driving styles, including variations in speed profiles and lane positioning. The model achieves an average 20% improvement in imitation scores compared to baseline methods that combine NMPC and neural networks but do not integrate NMPC into the differentiable training pipeline. Qualitative analysis reveals that the learned NMPC parameters provide interpretable insights into driving behaviors; for instance, the model adjusts weight parameters differently for left versus right curves to replicate specific acceleration and deceleration patterns observed in human demonstrations. Quantitative metrics, including absolute error and Z-scores relative to driver distributions, confirm that the learned models closely match the statistical distributions of human driving states. The significance of this work lies in its ability to provide safe, constraint-satisfying autonomous driving controllers that are both personalized and interpretable. By learning NMPC parameters directly from visual data, DriViDOC avoids the need for manual tuning and ensures that safety constraints are inherently satisfied during inference. The approach offers a robust solution for personalized autonomous driving, demonstrating that differentiable optimal control can effectively bridge the gap between high-level visual perception and low-level safe control, outperforming existing hybrid architectures in imitation fidelity.
Provenance
The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.
| Stage | Outcome | Tool | Model | Prompt | Attempts | Completed |
|---|---|---|---|---|---|---|
| discover | success | Crossref | — | — | 1 | 2026-06-25 |
| archive | success | unpaywall | — | — | 2 | 2026-06-26 |
| extract | success | cached | — | — | 2 | 2026-06-26 |
| clean | success | clean | — | — | 1 | 2026-06-26 |
| chunk | success | chunk | — | — | 1 | 2026-06-26 |
| embed | success | embed | Qwen/Qwen3-Embedding-8B | — | 1 | 2026-06-26 |
| enrich | success | openalex | — | — | 1 | 2026-06-26 |
| promote | success | — | — | — | 1 | 2026-06-25 |
| summarize | success | llm | qwen3.6-27b-prismaquant | summ-v5 | 1 | 2026-06-26 |
| tag | success | vector_similarity | — | — | 6 | 2026-06-26 |
| verify | success | — | — | — | 1 | 2026-06-26 |
Summary generated by qwen3.6-27b-prismaquant on 2026-06-26; verification: verified.
Topics
Ranked by relevance to this paper. Hover a topic for its definition.
Information type
What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).
- Methodological Resource: tool software
- Theoretical Contribution: computational model