基于改进StyleGAN的高分辨率可控肖像视频风格迁移网络

High Resolution Controllable Portrait Video Style Transfer Network Based on Improved StyleGAN

摘要: 肖像风格迁移是计算机视觉和图形学的一个重要领域。然而，当前很多肖像风格迁移算法在很大程度上未能捕捉到不同肖像风格的重要几何依赖。因为肖像风格迁移需要更注重特征的细化和风格的融合。与此同时，数据的稀缺性也是风格化的挑战之一，同时面向图像的方法在应用于视频时也会存在闪烁伪影等缺陷。针对肖像风格迁移，本文提出了基于改进StyleGAN的高分辨率可控肖像视频风格迁移算法HcpGAN (style transfer network for High-resolution Controllable Portrait video based on StyleGAN)。具体来说，HcpGAN由生成器和鉴别器组成，生成器采用内外双支路风格路径网络结构进行肖像风格迁移，分层式网络结构可以对风格程度进行可控与微调，通过扩张卷积对生成器第一层特征模块进行微调，解除了肖像输入固定裁剪限制。与此同时，在生成器的尾部集成处理视频帧的特征扭曲层，不使用额外的网络和光流预测，通过引入特征扭曲层直接对视频帧的时间一致性信息进行建模，从而输出时间序列平滑的风格化视频。在公开数据集上对比试验和消融实验显示，HcpGAN在当前肖像风格迁移算法中处于先进水平。

Abstract: Portrait style transfer is an important field in computer vision and graphics. However, many current algorithms largely fail to capture the important geometric dependencies of different portrait styles. Because the transfer of portrait style needs to pay more attention to the refinement of features and the fusion of styles. At the same time, the scarcity of data is also one of the challenges of stylization, and image style transfer methods also have defects such as flicker artifacts when applied to video. For the transfer of portrait style, this paper proposed a High-resolution Controllable Portrait video style transfer network based on StyleGAN (HcpGAN). Specifically, HcpGAN consists of a generator and discriminator. The generator adopts an internal and external double-branch style path network structure to carry out portrait style migration. The hierarchical network structure can control and fine-tune the style degree, and fine-tune the first-layer feature module of the generator by expanding convolution, which eliminates the restriction of fixed portrait input clipping. At the same time, the feature distortion layer of the video frame is integrated in the tail of the generator, and the temporal consistency of the video frames is directly modeled by introducing the feature distortion layer without using additional network and optical flow prediction, so as to output the time series smooth stylized video. Comparison and ablation experiments on open datasets show that HcpGAN is at an advanced level in the current portrait style transfer.