通过GRU和多头注意力机制增强学习型优化器的泛化能力

Enhancing the Generalization Ability of Learning-Based Optimizers through GRU and Multi-Head Attention Mechanisms

摘要: 近年来，利用机器学习(尤其是深度学习技术)解决数学问题的关注度持续上升。学习优化作为一种借助深度学习求解优化问题的方法，已吸引了越来越多的关注。在当前研究中，仅使用LSTM模型仍然是主流的选择，尽管LSTM模型可以更有效捕捉历史信息，但是其对信息之间的交互是不够充分的。因此我们选择了对其隐藏层的结果加上多头注意力机制，以强化信息之间的交融，并且将LSTM换为轻量化的GRU模型，故模型的参数量甚至是减少了的。实验结果表明，该算法不仅收敛速度更快，还展现出较强的泛化能力。

Abstract: In recent years, there has been a growing interest in using machine learning, particularly deep learning techniques, to address mathematical problems. Learning to Optimize, a method that leverages deep learning to solve optimization problems, has attracted increasing attention. In current research, the exclusive use of LSTM models remains the predominant choice. While LSTM models can effectively capture historical information, their ability to handle interaction between information is insufficient. Therefore, we propose adding a multi-head attention mechanism to the outputs of the hidden layer to enhance the fusion of information. We also replace the LSTM with a lightweight GRU model, resulting in an even reduction in the number of model parameters. Experimental results demonstrate that the algorithm not only achieves faster convergence but also exhibits strong generalization capabilities.