核梯度下降在回归问题中的泛化误差与谱偏置

Generalization Error and Spectral Bias of Kernel Gradient Descent in Regression

摘要: 本文主要研究使用核梯度下降求解最小二乘回归问题，并基于再生核希尔伯特空间的谱分解，给出了核梯度下降的泛化误差在各特征子空间上的模态误差关于优化时间的函数，这种分解方法有助于我们理解核梯度下降对回归函数在各特征空间上的分量的带偏置的学习，即谱偏置，以及噪声对各模态学习不同程度的影响。在使用核方法时，核函数的选择以及核函数超参数的选择尤为重要，我们的结果验证了任务与模型的对齐理论，这将帮助我们选择适合任务的核函数。由于宽神经网络的训练过程等价于使用神经正切核进行核梯度下降，本文的模态误差函数同样适用于此类网络。在推导本文主要结果时，我们用到了协方差算子的谱分解、矩阵指数函数的拉普拉斯逆变换以及样本协方差矩阵的各向异性局部律等方法，并用高斯核以及神经正切核在人工合成数据以及MNIST数据集上验证了本文的结果。

Abstract: This paper primarily investigates the use of kernel gradient descent for solving least squares regression problems. Based on the spectral decomposition of reproducing kernel Hilbert spaces, we present the generalization error of kernel gradient descent as a function of optimization time, specifically the mode error on each eigenspace. This decomposition helps us understand the biased learning of the kernel gradient descent on the components of the regression function in different eigenspaces—referred to as spectral bias—as well as the varying effects of noise on the learning of different modes. The choice of kernel function and its hyperparameters is crucial when applying kernel methods, and our results validate the task-model alignment theory, which aids in selecting appropriate kernel functions for specific tasks. Since the training process of wide neural networks is equivalent to kernel gradient descent using the neural tangent kernel, the mode error function derived in this paper is also applicable to such networks. In deriving the main results, we employ techniques such as the spectral decomposition of the covariance operator, the inverse Laplace transform of matrix exponential functions, and the anisotropic local law of the sample covariance matrix. The theoretical findings are validated on both synthetic data and the MNIST dataset using the Gaussian kernel and the neural tangent kernel.