Cooperative Deep Q-Learning With Q-Value Transfer for Multi-Intersection Signal Control

Ge, Hongwei; Song, Yumei; Wu, Chunguo; Ren, Jiankang; Tan, Guozhen · 2019 · OpenAlex-citations

DOI: 10.1109/access.2019.2907618

archive: archived pipeline: cataloged verified

Get this paper ↗ (DOI — opens at the source; we link to it, we don't host it)

Summary

This paper addresses the challenge of adaptive traffic signal control in multi-intersection systems, where complex intersection features, heterogeneous structures, and the need for dynamic coordination hinder traditional reinforcement learning methods. While deep reinforcement learning has shown promise for single intersections, existing multi-agent approaches often suffer from the "curse of dimensionality" in centralized control or lack sufficient coordination in decentralized independent modes. To overcome these limitations, the authors propose a Cooperative Deep Q-Network with Q-Value Transfer (QT-CDQN), designed to optimize regional traffic flow by balancing congestion across multiple intersections while accommodating varying intersection geometries. The methodology models a multi-intersection traffic network as a multi-agent reinforcement learning system, where each agent controls a single intersection using a deep Q-network. The state space is represented through discrete traffic state encoding, utilizing vehicle position and speed matrices derived from raw traffic data. A convolutional neural network (CNN) is employed to automatically extract features from this high-dimensional state information. To facilitate cooperation, the algorithm incorporates a Q-value transfer mechanism: each agent integrates the optimal Q-values of its neighboring agents from the previous time step into its own loss function during training. This allows agents to account for the influence of adjacent intersection actions. The framework also utilizes experience replay and a target network to ensure training stability. The reward function is defined as the reduction in average vehicle queue length at each intersection, incentivizing actions that alleviate congestion. Experimental studies were conducted on heterogeneous road networks, including combinations of three-legged and four-legged intersections, under both recurring and occasional congestion scenarios. The results demonstrate that QT-CDQN is competitive with state-of-the-art algorithms in terms of average queue length, average vehicle speed, and average waiting time. The method proved effective in balancing traffic flow across the region and adapting to dynamic traffic environments. Notably, the approach successfully handled heterogeneous intersection structures without requiring uniform phase configurations or state dimensions, addressing a significant limitation of prior Q-function transfer methods. The significance of this work lies in its ability to provide a scalable and versatile solution for multi-intersection signal control. By integrating Q-value transfer into a cooperative deep reinforcement learning framework, QT-CDQN avoids the computational intractability of centralized control and the coordination deficits of independent multi-agent systems. The use of CNNs for feature extraction eliminates the need for hand-crafted state representations, while the Q-value transfer mechanism ensures that local decisions contribute to global optimization. This approach offers a robust method for improving urban traffic efficiency and reducing congestion in complex, real-world road networks.

Provenance

The full processing record for this entry. Every stage of this paper's journey through the pipeline is logged — what ran, with which tool and model, how many attempts it took, and when it last completed.

Summary generated by qwen3.6-27b-prismaquant on 2026-06-26; verification: verified.

Topics

Ranked by relevance to this paper. Hover a topic for its definition.

• traffic density