基于Polyak步长的动量方法

Polyak Step-Size for Momentum Method

摘要: 近年来，动量方法广泛地应用在机器学习训练中。本文基于Polyak步长和移动平均动量(MAG)方法提出了一个新的动量方法(LAGP)，并将其与随机梯度结合，提出SLAGP方法。建立了LAGP方法在半强凸条件下的线性收敛性，以及SLAGP算法在半强凸条件下的线性收敛性。数值实验表明LAGP和SLAGP与其他流行算法相比有明显优势。

Abstract: Recently, momentum methods have been widely adopted in training machine learning. In this paper, based on the Polyak step-size and the Moving Average Gradient (MAG) method, a new momentum method (LAGP) is proposed. By combining it with the stochastic gradient, the SLAGP method is developed. The linear convergence of the LAGP method under the semi-strongly convex condition, and the linear convergence of the SLAGP algorithm under the semi-strongly convex condition are established. Numerical experiments show that LAGP and SLAGP have significant advantages compared with other popular algorithms.