基于梯度范数感知最小化的去中心化联邦学习算法

A Decentralized Federated Learning Algorithm Based on Gradient Norm Aware Minimization

摘要: 去中心化联邦学习是通过一组设备执行隐私保护的分布式学习，它有效降低了中心化联邦学习的通信成本和信息泄露风险。然而，设备之间的非独立同分布数据会影响模型效果。为了解决这个问题，几乎所有的算法都利用经验风险最小化作为局部优化器，但这很容易造成客户端本地训练过拟合，造成算法全局模型的泛化能力下降。本文利用梯度范数感知最小化，提出基于梯度范数感知最小化的去中心化联邦学习算法，使全局模型损失函数的表面更加平滑，提升模型的泛化能力。

Abstract: Decentralized Federated Learning performs privacy-preserving distributed learning across a group of devices, reducing the communication costs and information leakage risks associated with centralized federated learning. However, the non-independent and identically distributed (Non-IID) data among devices can negatively impact the model’s performance. To address this issue, most algorithms adopt empirical risk minimization as the local optimizer, which often leads to overfitting during local client training and results in decreased generalization ability of the global model. This paper proposes a Decentralized Federated Learning Algorithm based on Gradient Norm-Aware Minimization, which smooths the loss surface of the global model and enhances its generalization performance.