Reinforcement Learning

Redmill, Keith; Zhang, Zhihao; Yurtsever, Ekim · 2024 · ROSA P / Carnegie Mellon University. Traffic21 Institute. Safety21 University Transportation Center (UTC)

archive: archived pipeline: cataloged verified

Get this paper ↗ (full text — opens at the source; we link to it, we don't host it)

Summary

This research addresses the challenge of developing autonomous driving systems capable of navigating complex traffic environments, specifically focusing on the limitations of existing Deep Reinforcement Learning (DRL) controllers. Current DRL approaches often rely on simple traffic patterns and struggle with delayed, long-term rewards, which compromises their generalizability and safety in intricate real-world scenarios. To overcome these issues, the authors propose a hierarchical DRL framework that decomposes complex decision-making into manageable subtasks, enhancing both exploration potential and control precision. The methodology formulates highway driving as a Markov Decision Process (MDP) within a simulation environment. The proposed system utilizes a two-level architecture: a high-level controller responsible for strategic decisions (lane changes and speed adjustments) using long-term delayed rewards, and a low-level controller handling immediate longitudinal and lateral control via short-term instantaneous rewards. The authors employ a two-step training process, first training the high-level controller using a model-based motion planner and a critic function, then using the trained high-level controller to facilitate the training of the low-level controller. The experimental design includes a specific "trap" scenario where the ego vehicle must navigate around slow-moving traffic vehicles to test the agent's ability to explore long-term benefits and escape suboptimal states. Traffic vehicles are controlled using the Intelligent-Driver Model (IDM) and the MOBIL model for realistic behavior. Simulation experiments demonstrate the superiority of the hierarchical DRL controller over single-level DRL and h-DQN baselines. The hierarchical approach exhibited a higher success rate in navigating out of low-speed traffic traps, achieved higher average rewards per episode, and maintained consistently higher average speeds. The results indicate that the hierarchical structure allows for more effective exploration of the environment, enabling the agent to identify and execute optimal long-term strategies, such as overtaking maneuvers, while maintaining safe and efficient driving dynamics. The low-level controller successfully executed fine-grained control maneuvers, ensuring stability during complex interactions. The significance of this work lies in its contribution to the robustness and adaptability of automated driving systems. By addressing the exploration limitations of standard DRL in complex scenarios, the hierarchical framework offers a more interpretable and efficient solution for highway driving. The study highlights the importance of separating long-term strategic planning from immediate control actions to improve safety and performance. The authors provide open-source code for their implementation, facilitating further research and validation in the field of autonomous vehicle control. This approach represents a step toward more reliable autonomous systems capable of handling the unpredictable and dynamic conditions of real-world traffic.

Key finding

The hierarchical deep reinforcement learning controller outperformed single-level controllers in simulation by achieving higher success rates, average speeds, and rewards when navigating complex highway traffic scenarios.

Methodology

simulator

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed. Discovered via bulk_ingest_rosap on 2026-05-23 (6 acquisition events logged).

Summary generated by qwen3.6-27b-prismaquant on 2026-06-10; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• situational awareness

Information type

What kind of knowledge this paper contributes, grouped by family — independent of topic (what it is about) and method (how it was studied).

• Theoretical Contribution: computational model