带有先验的语音驱动三维人脸动画生成方法

Speech Driven 3D Facial Animation Generation Method with Prior Knowledge

摘要: 语音驱动的三维人脸生成是计算机视觉和图形学中一个非常有吸引力的研究课题。除了有趣之外，它还有广泛的应用，例如游戏动画、3D视频通话和AR/MR的3D化身。由于人脸运动的复杂性和不确定性，以往方法生成的结果有唇形不准确、面部动态性不佳的缺点。不同于以往一阶段的方法，我们使用一种新的两阶段的方法，在模型训练的第一阶段我们使用变分自动编码器将高维的复杂的面部映射进低维的空间，充分学习人脸运动先验。在第二阶段，Transformer根据输入的语音信号在学习到的人脸先验的基础上进行潜在代码查询，以回归的方式生成面部运动序列。这样可以降低生成面部动画的难度，减少了映射的模糊，可以在任意指定音频上得到生动的人脸说话动画，经验证我们的方法与先进的方法相比在唇形和脸部动态性上取得优势。

Abstract: Speech-driven 3D facial animation is a very attractive research topic in computer vision and graphics. In addition to being interesting, it has a wide range of applications, such as game anima-tion, 3D video calls, and 3D avatars of AR/MR. Due to the complexity and uncertainty of facial movements, previous methods have drawbacks such as inaccurate lip shape and poor facial dynamics. Unlike previous methods, we use a new two-stage approach. In the first stage of model training, we use a variational autoencoder to map high-dimensional complex faces into low-dimensional space, fully learning facial motion priors. In the second stage, the Transformer performs latent code queries based on the learned facial prior based on the input speech signal, and generates facial motion sequences through regression. This can reduce the difficulty of generating facial animation, reduce mapping blur, and obtain vivid facial speech animations on any specified audio. It has been verified that our method has advantages in lip shape and facial dynamics compared to advanced methods.