基于混合域注意力的深度强化学习交叉口信号控制方法

An Intersection Signal Control Method with Deep Reinforcement Learning Based on Mixed Domain Attention

摘要: 针对强化学习智能体对微观交通状态感知能力有限的问题，本文提出了一种基于混合域注意力的深度强化学习交叉口信号控制算法3DQN_MDAM。首先，为减少存储开销，设计了一种轻量的混合域注意力模块(Mixed Domain Attention Module, MDAM)，仅使用少量的参数就能实现自适应地调整交通状态特征图中通道之间及空间位置之间权重的功能。然后，在现有基于双深度决斗Q网络(Double Dueling DQN, 3DQN)算法模型的基础上通过引入MDAM，使智能体自动地聚焦于对当前控制任务更为重要的交通状态信息，以增强智能体的状态感知能力。最后，利用仿真平台SUMO (Simulation of Urban Mobility)进行实验。实验结果显示，在低、中、高三种不同交通流条件下，3DQN_MDAM相比3DQN在各项指标上均得到改善，其中车辆平均等待时间分别缩短了20%、20%、17.6%。与其它常用的基准算法相比，3DQN_MDAM在各项指标上均得到最好的控制效果。

Abstract: Aiming at the problem that reinforcement learning agents have limited perception of microscopic traffic conditions, an intersection signal control algorithm 3DQN_MDAM with deep reinforcement learning based on mixed domain attention is proposed. Firstly, to reduce storage overhead, a lightweight mixed domain attention module (MDAM) is designed, which can adaptively adjust the weights between channels and spatial positions in the traffic state feature map with only a small number of parameters. Then, based on the existing 3DQN algorithm model, by introducing MDAM, the agent automatically focuses on traffic status information that is more important to the current control task, in order to enhance the agent’s state perception ability. Finally, experiments were conducted using the simulation platform SUMO (Simulation of Urban Mobility). The experimental results show that under three different traffic flow conditions of low, medium, and high, 3DQN_MDAM has improved in various indicators compared to 3DQN, with average waiting time of vehicles reduced by 20%, 20%, and 17.6%, respectively. Compared with other commonly used benchmark algorithms, 3DQN_MDAM achieved the best control effect on all indicators.