深度强化学习模型轻量化算法研究
Research on Lightweight Algorithms for Deep Reinforcement Learning
摘要: 针对深度强化学习网络难以部署到资源受限终端设备的问题，本文提出一种深度神经网络优化压缩算法。该算法引入倒残差模块作为主干网络，实现网络的轻量化；采用基于响应的知识蒸馏，以动作策略为蒸馏目标，弥补网络轻量化造成的精度损失；采用基于特征的知识蒸馏，对网络中间层的特征向量进行蒸馏，进一步提升网络精度。实验结果表明，轻量化后的网络参数量为19.79M，参数量为原网络的59.8%，性能提升约12.1%，且在网络轻量化的同时，提升了模型表现，验证了所提算法的有效性。
Abstract: In response to the difficulty of deploying deep reinforcement learning networks on resource- constrained terminal devices, a deep neural network optimization compression algorithm is proposed in this paper. This algorithm introduces an inverse residual module as the backbone network to achieve the lightweight of network; adopts response-based knowledge distillation, with action strategy as the distillation target, to make up for the accuracy loss caused by the lightweight of network; adopts feature-based knowledge distillation to distill the feature vectors in the middle layer of the network, further improving network accuracy. Experimental results show that the parameter size of the lightweight network is 19.79M, the parameter size is 59.8% of the original network, the performance is improved by about 12.1%, and the model performance is improved while the network is lightweight, verifying the effectiveness of the proposed algorithm.
文章引用：安天一, 李宁, 王超. 深度强化学习模型轻量化算法研究[J]. 计算机科学与应用, 2023, 13(4): 779-788. https://doi.org/10.12677/CSA.2023.134077

参考文献

[1] Mao, H., Schwarzkopf, M., Venkatakrishnan, S.B., et al. (2019) Learning Scheduling Algorithms for Data Processing Clusters. Proceedings of the ACM Special Interest Group on Data Communication, Beijing, 19-23 August 2019, 270-288. [Google Scholar] [CrossRef] 
[2] Long, P., Fan, T., Liao, X., et al. (2018) Towards Opti-mally Decentralized Multi-Robot Collision Avoidance via Deep Reinforcement Learning. 2018 IEEE International Con-ference on Robotics and Automation (ICRA), Brisbane, 21-25 May 2018, 6252-6259. [Google Scholar] [CrossRef] 
[3] Li, D., Zhao, D., Zhang, Q., et al. (2019) Reinforcement Learn-ing and Deep Learning Based Lateral Control for Autonomous Driving. IEEE Computational Intelligence Magazine, 14, 83-98. [Google Scholar] [CrossRef] 
[4] Liao, X., Li, W., Xu, Q., et al. (2020) Iteratively-Refined Interactive 3D Medical Image Segmentation with Multi-Agent Reinforcement Learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, 13-19 June 2020, 9394-9402. [Google Scholar] [CrossRef] 
[5] Vinyals, O., Babuschkin, I., Czarnecki, W.M., et al. (2019) Grandmaster Level in StarCraft II Using Multi-Agent Reinforcement Learning. Nature, 575, 350-354. [Google Scholar] [CrossRef] [PubMed]
[6] Ye, D., Chen, G., Zhao, P., et al. (2020) Supervised Learning Achieves Human-Level Performance in Moba Games: A Case Study of Honor of Kings. IEEE Transactions on Neural Networks and Learning Systems, 33, 908-918. [Google Scholar] [CrossRef] 
[7] 李琛, 黄炎焱, 张永亮, 等. Actor-Critic框架下的多智能体决策方法及其在兵棋上的应用[J]. 系统工程与电子技术, 2021, 43(3): 755-762.
[8] Wang, P. and Goertzel, B. (2007) Introduction: Aspects of Artificial General Intelligence. Proceedings of the 2007 Conference on Advances in Arti-ficial General Intelligence: Concepts, Architectures and Algorithms, Washington DC, June 2007, 1-16.
[9] Bellemare, M.G., Naddaf, Y., Veness, J., et al. (2013) The Arcade Learning Environment: An Evaluation Platform for General Agents. Journal of Artificial Intelligence Research, 47, 253-279. [Google Scholar] [CrossRef] 
[10] Coulom, R. (2007) Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search. Computers and Games: 5th International Conference, Turin, 29-31 May 2006, 72-83. [Google Scholar] [CrossRef] 
[11] Ye, W., Liu, S., Kurutach, T., et al. (2021) Mastering Atari Games with Limited Data. Annual Conference on Neural Information Pro-cessing Systems 2021, NeurIPS 2021, 6-14 December 2021, 25476-25488.
[12] Sandler, M., Howard, A., Zhu, M., et al. (2018) MobileNetV2: Inverted Residuals and Linear Bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, 18-22 June 2018, 4510-4520. [Google Scholar] [CrossRef] 
[13] 赵冬斌, 邵坤, 朱圆恒, 等. 深度强化学习综述: 兼论计算机围棋的发展[J]. 控制理论与应用, 2016, 33(6): 701-717.
[14] Li, Y. (2017) Deep Reinforcement Learning: An Over-view. ArXiv: 1701.07274.
[15] Arulkumaran, K., Deisenroth, M.P., Brundage, M., et al. (2017) A Brief Survey of Deep Reinforcement Learning. ArXiv: 1708.05866.
[16] Silver, D., Huang, A., Maddison, C.J., et al. (2016) Mastering the Game of Go with Deep Neural Networks and Tree Search. Nature, 529, 484-489. [Google Scholar] [CrossRef] [PubMed]
[17] Silver, D., Schrittwieser, J., Simonyan, K., et al. (2017) Mastering the Game of Go without Human Knowledge. Nature, 550, 354-359. [Google Scholar] [CrossRef] [PubMed]
[18] Schrittwieser, J., Antonoglou, I., Hubert, T., et al. (2020) Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model. Nature, 588, 604-609. [Google Scholar] [CrossRef] [PubMed]
[19] Hinton, G., Vinyals, O. and Dean, J. (2015) Distilling the Knowledge in a Neural Network. ArXiv: 1503.02531.
[20] Romero, A., Ballas, N., Kahou, S.E., et al. (2014) Fitnets: Hints for Thin Deep Nets. ArXiv: 1412.6550.
[21] Iandola, F.N., Han, S., Moskewicz, M.W., et al. (2016) SqueezeNet: AlexNet-Level Accuracy with 50x Fewer Parameters and < 0.5 MB Model Size. ArXiv: 1602.07360.
[22] Howard, A.G., Zhu, M., Chen, B., et al. (2017) Mobilenets: Efficient Convolutional Neural Networks for Mobile Vision Applica-tions. ArXiv: 1704.04861.
[23] Howard, A., Sandler, M., Chu, G., et al. (2019) Searching for MobileNetV3. Proceed-ings of the IEEE/CVF International Conference on Computer Vision, Seoul, 27 October-2 November 2019, 1314-1324. [Google Scholar] [CrossRef] 
[24] Hu, J., Shen, L. and Sun, G. (2018) Squeeze-and-Excitation Net-works. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, 18-23 June 2018, 7132-7141. [Google Scholar] [CrossRef] 
[25] Zhang, X., Zhou, X., Lin, M., et al. (2018) ShuffleNet: An Ex-tremely Efficient Convolutional Neural Network for Mobile Devices. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, 18-23 June 2018, 6848-6856. [Google Scholar] [CrossRef] 
[26] Ma, N., Zhang, X., Zheng, H.T., et al. (2018) ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design. Proceedings of the European Conference on Computer Vi-sion (ECCV), Munich, 8-14 September 2018, 116-131. [Google Scholar] [CrossRef] 
[27] Goodfellow, I., Pouget-Abadie, J., Mirza, M., et al. (2020) Generative Adversarial Networks. Communications of the ACM, 63, 139-144. [Google Scholar] [CrossRef] 
[28] Zoph, B. and Le, Q.V. (2016) Neural Architecture Search with Reinforce-ment Learning. ArXiv: 1611.01578.
[29] Kipf, T.N. (2020) Deep Learning with Graph-Structured Representations. Ph.D. Thesis, Universiteit van Amsterdam, Amsterdam, 164 p.
[30] Schapire, R.E. (1990) The Strength of Weak Learn Ability. Machine Learning, 5, 197-227. [Google Scholar] [CrossRef] 
[31] Fu, M.C. (2019) Simulation-Based Algorithms for Markov Decision Processes: Monte Carlo Tree Search from Alphago to Alphazero. Asia-Pacific Journal of Operational Research, 36, Ar-ticle ID: 1940009. [Google Scholar] [CrossRef] 
[32] Rosin, C.D. (2011) Multi-Armed Bandits with Episode Context. Annals of Mathematics and Artificial Intelligence, 61, 203-230. [Google Scholar] [CrossRef] 
[33] Kocsis, L. and Szepesvári, C. (2006) Bandit Based Monte-Carlo Planning, Machine Learning. ECML 2006: 17th European Conference on Machine Learning, Berlin, 18-22 September 2006, 282-293. [Google Scholar] [CrossRef] 
[34] Kaiser, L., Babaeizadeh, M., Milos, P., et al. (2019) Model-Based Re-inforcement Learning for Atari. ArXiv: 1903.00374.
[35] Kielak, K.P. (2020) Do Recent Advancements in Model-Based Deep Reinforcement Learning Really Improve Data Efficiency? ArXiv: 2003.10181.
[36] Laskin, M., Srinivas, A. and Abbeel, P. (2020) CURL: Contrastive Unsupervised Representations for Reinforcement Learning. International Confer-ence on Machine Learning PMLR, Vienna, 13-18 July 2020, 5639-5650.
[37] Kostrikov, I., Yarats, D. and Fergus, R. (2020) Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels. ArXiv: 2004.13649.
[38] Schwarzer, M., Anand, A., Goel, R., et al. (2020) Data-Efficient Reinforcement Learning with Self-Predictive Representations. ArXiv: 2007.05929.