带Group Lasso正则项的Pi-Sigma神经网络在线梯度算法研究

Online Gradient Algorithm for Pi-Sigma Neural Networks with Smooth Group Lasso Regularizer

摘要: Pi-Sigma神经网络是隐层带有求和神经元，输出层带有求积神经元的一种前馈神经网络，该网络具有较强的非线性映射能力。在误差函数中添加正则项是神经网络常用的优化方法，和传统的L2、L1/2正则项相比Group Lasso正则项可以在组级别上消除不必要的权值，具有良好的稀疏效果。众所周知，利用梯度法进行权值更新的学习方式有两种：一种是批处理学习算法，另一种是在线学习算法。本文提出带Group Lasso正则项的在线梯度学习算法来训练Pi-Sigma神经网络。最后，数值实验结果表明改进后的算法收敛速度更快并且具有较好的泛化性能。

Abstract: Pi-Sigma neural network is a feedforward neural network with summation neurons in hidden layer and quadrature neurons in output layer, which has strong nonlinear mapping ability. Adding regular terms to the error function is a common optimization method for neural networks. Compared with the traditional L2 and L1/2 regular terms, the Group Lasso regular terms can eliminate unnecessary weights at the group level and have a good sparse effect. As is known to all, there are two learning methods for weight update using gradient method: one is batch learning algorithm; the other is online learning algorithm. This paper proposes an online gradient learning algorithm with Group Lasso regularized terms to train Pi-Sigma neural networks. Finally, numerical results show that the improved algorithm converges faster and has better generalization performance.