基于ADMM算法的鲁棒正则化问题求解方法

New Numerical Methods for Robust Regularization Problem Based on Alternating Direction Method of Multipliers

摘要: 线性回归是机器学习中的重要模型，但是过大的数据量会使线性回归模型陷入过拟合。正则化技术是解决过拟合问题的主要方法。标准的正则化模型是运用线性回归项和正则项协同作用以达到保证分类质量和降低陷入过拟合风险的目的。但是当训练样本存在扰动时，标准的正则化模型得到的分类器会因为数据集存在扰动而表现得不稳定。针对该缺点，本文提出了两种鲁棒正则化模型：1) 随机鲁棒正则化：将残差的数学期望和正则项结合；2) 最坏情况鲁棒正则化：将最坏情况的残差和正则项结合。然后利用交替方向法(交替方向法，ADMM)求得鲁棒正则化模型的最优解。数值实验显示：当训练存在扰动的数据集时，随机鲁棒正则化和最坏情况鲁棒正则化得到的分类器具有很好的鲁棒性，而标准正则化方法得到的分类器波动很大。

Abstract: Linear regression is an important model in machine learning, but too large data will make the linear regression model fall into over-fitting problem. Regularization is the main method to solve the over-fitting problem. The standard regularization model is the method that linear re-gression co-operates with regularization to ensure the quality of classification and reduce the risk of falling into over-fitting. However, the classifier obtained by the standard regularization model will be unstable when the training samples are disturbed. To solve this problem, we pro-pose two kinds of robust regularization models: 1) Stochastic robust regularization: combining the expectation of residual with the regularization; 2) Worst case robust regularization: com-bining the worst case residual with the regularization. Then, we use Alternating Direction Method of Multipliers (ADMM) algorithm to get the optimal solution of the robust regularization model. Numerical experiments show that the classifiers obtained by stochastic and worst-case robust regularization have good robustness when training disturbed data sets, but the classifi-ers obtained by standard regularization method fluctuate greatly.