基于PID控制更新的Sarsa强化学习算法及应用

Sarsa Reinforcement Learning Algorithm and Application Based on PID Control Update

摘要: 针对强化学习中Sarsa算法收敛速度慢且效果不稳定的问题，考虑到PID控制操作简单且鲁棒性高，提出基于PID控制优化的Sarsa算法，即Pid_Sarsa。其主要思想是将Sarsa算法中Q值的迭代方式改进为三项之和，分别对应PID控制中的比例、积分和微分，体现了对当前、过去和未来的误差进行控制的思想，理论上提高了样本利用率。为了对比Pid_Sarsa算法与Sarsa和n_Sarsa(取n = 5)两种传统算法的效果，选择悬崖寻路这一经典路径规划游戏作为算例，实验表明：Pid_Sarsa算法收敛速度更快、效果更稳定，且得到的路径安全程度比Sarsa算法高2.38%，比5步Sarsa算法高4.76%。

Abstract: In view of the problem of slow convergence speed and unstable effect of Sarsa algorithm in rein-forcement learning, considering the simple operation and high robustness of PID control, we pro-pose a Sarsa algorithm based on PID control optimization, that is, Pid_Sarsa. The main idea is that improve the iterative way of Q values in Sarsa algorithm to the sum of three terms, corresponding to the proportions, integrals and differentiation in PID control, which reflect the idea of controlling current, past and future errors, and theoretically improves sample utilization. In order to compare the effects of the Pid_Sarsa algorithm with the two traditional algorithms of Sarsa and n_Sarsa (n = 5), the classic path planning game of cliff walking is selected as an example, and the experiments show that the Pid_Sarsa algorithm converges faster and the effect is more stable, and the obtained path security degree is 2.38% higher than that of Sarsa algorithm and 4.76% higher than that of 5-step Sarsa algorithm.