基于滑动窗口的自适应学习率优化方法

Sliding-Window Based Adaptive Learning Rate Optimization Method

摘要: 针对深度神经网络训练中梯度噪声干扰导致参数更新偏离和高曲率区域参数振荡的问题，给出一种在随机优化中能够快速响应局部梯度噪声变化的自适应学习率优化方法(SW-Adam)。该方法设计了滑动窗口方差估计和方差感知学习率调整两个协同机制：一方面，采用Welford在线算法对窗口内梯度赋予均匀权重1/k，将方差估计的响应延迟从传统指数加权移动平均(EMA)的$O\left(1/\left(1-\beta \right)\right)$ (约1000步)降至O(k) (k为窗口长度，通常取50)；另一方面，通过指数衰减函数建立学习率与方差估计的负相关关系，在高噪声区域自动降低学习率以抑制振荡，在低噪声区域提高学习率以加速收敛。在60维带噪声旋转Rastrigin函数上的初步数值实验结果表明，SW-Adam的最终函数值为130.80，相比AdamW降低77.6%，相比Shampoo降低88.1%；方差估计波动系数相比Adam降低8.6%；虽然SW-Adam的单步计算耗时略高于Adam，但其达到相同目标函数值所需的总迭代次数更少；Wilcoxon秩和检验证实与各基线算法的性能差异具有统计显著性(p < 0.001)。消融实验表明三个组件各具独立贡献，其中滑动窗口方差估计贡献最大(移除后性能下降202.3%)，方差感知指数衰减和学习率裁剪分别提供关键的信号转化和边界稳定保障。在CIFAR-10 + ResNet-18深度学习任务上，SW-Adam取得93.47%的测试准确率，泛化差距为3.21个百分点，验证了该方法在深度学习任务上的泛化能力。滑动窗口方差估计机制能够较好地反映局部梯度噪声水平，与方差感知学习率调整策略协同工作，在保持计算高效的同时有效改善收敛性能。

Abstract: To address the problems of parameter update deviation caused by gradient noise and parameter oscillation in high-curvature regions during deep neural network training, this study proposes an adaptive learning-rate optimization method, termed SW-Adam, that can rapidly respond to local variations in gradient noise under stochastic optimization. The method incorporates two coordinated mechanisms: sliding-window variance estimation and variance-aware learning rate adjustment. On the one hand, the Welford online algorithm is employed to assign a uniform weight of 1/k to gradients within the window, thereby reducing the response delay of variance estimation from $O\left(1/\left(1-\beta \right)\right)$ in the conventional exponential moving average (EMA) approach (approximately 1000 steps) to O(k), where k denotes the window length and is typically set to 50. On the other hand, an exponential decay function is used to establish a negative correlation between the learning rate and the variance estimate, so that the learning rate is automatically decreased in high-noise regions to suppress oscillations and increased in low-noise regions to accelerate convergence. Preliminary numerical experiments on the 60-dimensional noisy rotated Rastrigin function show that SW-Adam achieves a final function value of 130.80, representing a 77.6% reduction compared to AdamW and an 88.1% reduction compared to Shampoo; the variance estimation coefficient of variation decreases by 8.6% compared to Adam; although the single-step computation time of SW-Adam is slightly higher than that of Adam, it requires fewer total iterations to achieve the same objective function value; Wilcoxon rank-sum test confirms that the performance differences with all baseline algorithms are statistically significant (p < 0.001). Ablation experiments confirm the independent contribution of each component: the sliding-window variance estimation is the primary contributor (performance degrades by 202.3% upon removal), while the variance-aware exponential decay and learning rate clipping provide essential signal transformation and boundary stabilization, respectively. On the CIFAR-10 image classification task with ResNet-18, SW-Adam achieves a test accuracy of 93.47% with a generalization gap of 3.21 percentage points, demonstrating its generalization capability on practical deep learning tasks. The sliding-window variance estimation mechanism can effectively reflect local gradient noise levels, working synergistically with the variance-aware learning rate adjustment strategy to improve convergence performance while maintaining computational efficiency.