Autonomous On-ramp Merge Strategy Using Deep Reinforcement Learning in Uncertain Highway Environment

Wu, Sifan; Tian, Daxin; Zhou, Jianshan; Duan, Xuting; Sheng, Zhengguo; Zhao, Dezong · 2022 · OpenAlex-citations

DOI: 10.1109/icus55513.2022.9986560

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the challenge of autonomous on-ramp merging in uncertain highway environments, a complex scenario where traditional rule-based models often fail due to their inability to handle dynamic traffic conditions and sensor noise. The authors propose a Deep Reinforcement Learning (DRL) approach to enable autonomous vehicles to merge safely and efficiently without relying on vehicle-to-everything (V2X) communications. Specifically, they introduce a modified algorithm called Recurrent-based Twin Delayed Deep Deterministic Policy Gradient (RTD3), which integrates Long Short-Term Memory (LSTM) networks into the actor network of the standard TD3 algorithm. This modification allows the agent to process temporal sequences of state information, improving its ability to make decisions based on historical context rather than just instantaneous observations. The study was conducted using the Simulation of Urban Mobility (SUMO) platform, modeling a realistic highway segment based on I-80 with a 500-meter on-ramp. The reinforcement learning agent controls the longitudinal acceleration of the merging vehicle within a defined control zone, using a state space that includes distances and velocities of surrounding vehicles, as well as the ego vehicle’s position, speed, and acceleration. To simulate real-world conditions, the authors injected Gaussian noise into the sensor data for surrounding vehicles. The reward function penalizes collisions, stopping, and deviations from optimal speed, while rewarding successful merges. The RTD3 agent was compared against a standard TD3 agent across noise-free, mid-level noise (5%), and high-level noise (10%) environments. The results demonstrate that the RTD3 agent significantly outperforms the standard TD3 agent in both learning efficiency and robustness. In noise-free environments, RTD3 converged 600 episodes faster than TD3, achieving comparable average speeds (20.9 m/s vs. 20.8 m/s). More critically, in mid-level noise environments, RTD3 maintained a 99.1% collision-free rate with an average speed of 20.6 m/s, whereas TD3 performance degraded significantly, dropping to an 87% collision-free rate and an average speed of 19.2 m/s. Although RTD3 required more training time per episode due to the added complexity of the LSTM layer, its ability to handle partial observations and sensor noise made it far more suitable for realistic, stochastic traffic scenarios. The significance of this work lies in its demonstration that incorporating temporal memory into DRL algorithms enhances the generalization and stability of autonomous driving strategies in uncertain environments. By addressing the limitations of standard DDPG and TD3 algorithms regarding Q-value overestimation and sensitivity to noise, the proposed RTD3 method provides a more reliable solution for on-ramp merging. This contributes to the development of fully autonomous vehicles capable of operating safely in complex, real-world traffic conditions without extensive infrastructure support.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-prismaquant on 2026-06-26; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• situational awareness