基于Wasserstein距离作为GAN的优化目标提高其训练稳定性的理论研究

A Theoretical Study of Improving the Training Stability of GAN Based on Wasserstein Distance as Optimization Objective

摘要: 生成对抗网络(Generative adversarial Nets，以下简称GANs)因其在图像生成等领域的成功应用而备受关注。然而，其训练的不稳定性一直是一个难以解决的问题，训练过程常常受到模式崩溃、梯度消失和优化不稳定的困扰。一般提高GANs训练稳定性的方法有替代损失函数、梯度惩罚、谱归一化、批量归一化和架构改进等方法。但是这些研究大多缺乏理论基础，未给出相对完善的理论证明，本论文的目标是深入理解基于Wasserstein距离训练GANs的不稳定性，提供较为完整的理论证明。并探讨了进一步改进WGAN训练稳定性的策略，如梯度惩罚(WGAN-GP)，以提高WGAN训练的稳定性和泛化能力。本文的主要研究内容如下：第一部分：分析了WGAN通过最小化Wasserstein距离(简称W距离)代替传统的Jensen-divergence (简称JS散度)，避免了梯度消失问题。其关键优势在于采用了1-Lipschitz连续的判别器，确保了在训练过程中生成器能够从判别器获得有效梯度。其次，证明了W距离相较于其他距离或者散度对于概率分布序列具有良好的连续性和收敛性。第二部分：通过引入W距离替代原来两个分布之间的JS散度，从理论上改善了GANs训练的稳定性。然而，WGAN的实现仍面临挑战，如权重裁剪导致的容量利用不足和梯度消失问题。为此，基于W距离，Gulrajani等人提出了梯度惩罚(WGAN-GP)来满足Lipschitz约束，以进一步提高训练稳定性。但是大多文献直接给出梯度惩罚常数为1，并未给出具体证明，在本文中给出了证明。

Abstract: Generative adversarial networks (GANs) have attracted much attention due to their successful applications in fields such as image generation. However, the instability of their training has always been a difficult problem to solve, and the training process is often plagued by mode collapse, gradient vanishing, and optimization instability. General methods to improve the stability of GANs training include alternative loss functions, gradient penalties, spectral normalization, batch normalization, and architectural improvements. However, most of these studies lack a theoretical basis and do not provide a relatively complete theoretical proof. The goal of this paper is to deeply understand the instability of GANs training based on Wasserstein distance and provide a relatively complete theoretical proof. It also explores strategies to further improve the stability of WGAN training, such as gradient penalty (WGAN-GP), to improve the stability and generalization ability of WGAN training. The main research contents of this paper are as follows: Part I: WGAN is analyzed to avoid the gradient vanishing problem by minimizing the Wasserstein distance (W distance for short) instead of the traditional Jensen-divergence (JS divergence for short). Its key advantage is the use of 1-Lipschitz continuous discriminator, which ensures that the generator can obtain effective gradients from the discriminator during training. Secondly, it is proved that W distance has good continuity and convergence for probability distribution sequences compared with other distances or divergences. Part II: By introducing W distance to replace the JS divergence between the original two distributions, the stability of GANs training is theoretically improved. However, the implementation of WGAN still faces challenges, such as insufficient capacity utilization and gradient vanishing problems caused by weight clipping. To this end, based on W distance, Gulrajani et al. proposed a gradient penalty (WGAN-GP) to meet the Lipschitz constraint to further improve training stability. However, most literature directly gives the gradient penalty constant as 1, without giving a specific proof, which is given in this article.