基于三维场景的语义感知风格迁移

Semantic-Aware Style Transfer Based on 3D Scenes

摘要: 随着电影和游戏行业的快速发展，三维场景的创建与编辑方法不断优化，逐渐向更高效、更便捷的方向发展。相比传统的网格和点云表示方法，三维高斯提供了一种更灵活且高效的三维场景表示方式，能够在保证高质量渲染效果的同时，生成逼真的新视角图像。然而，现有的三维高斯模型在风格化方面仍存在局限性，难以满足创意设计和艺术表达的需求。因此，如何在保持三维结构信息的同时，实现高质量的风格迁移，成为一个值得深入研究的问题。针对这一问题，本文提出了一种基于三维高斯的语义风格迁移方法。首先，通过多视角图像训练三维高斯模型，并在这些图像上进行风格迁移，以确保三维模型的风格一致性和结构完整性。具体而言，我们利用LSeg模型对内容图像和风格图像进行语义分割，提取对应区域后，基于图像复杂度自适应确定聚类类别数量，在颜色空间采用K均值聚类进行分割，并以聚类区域面积筛选有效的结构信息。随后，通过语义匹配进行风格迁移，并结合WCT进行风格融合，最终使用VGG解码器生成风格化图像。实验结果表明，本文方法在风格质量、结构保持性和多视角一致性方面均优于现有方法，为三维艺术创作提供了更高质量的风格迁移效果和更强的可控性。

Abstract: With the rapid development of the film and gaming industries, methods for creating and editing 3D scenes have been continuously optimized, evolving toward greater efficiency and convenience. Compared to traditional representations based on meshes and point clouds, 3D Gaussian Splatting provides a more flexible and efficient way to represent 3D scenes, enabling high-quality novel view synthesis while maintaining superior rendering performance. However, existing 3D Gaussian models still have limitations in stylization, making it difficult to meet the demands of creative design and artistic expression. Therefore, achieving high-quality style transfer while preserving 3D structural information remains a challenging research problem. To address this issue, we propose a semantic style transfer method based on 3D Gaussians. First, a 3D Gaussian model is trained using multi-view images, and style transfer is performed on these images to ensure consistency and structural integrity in the final 3D model. Specifically, we utilize the LSeg model for semantic segmentation of content and style images. After extracting corresponding regions, we adaptively determine the number of clusters based on image complexity and apply K-means clustering in the color space to segment the images. The clustered regions are then filtered based on their area to retain essential structural information. Subsequently, style transfer is performed using semantic matching, and style fusion is achieved with the Whitening and Coloring Transform (WCT). Finally, a VGG-based decoder generates the stylized images. Experimental results demonstrate that our method outperforms existing approaches in terms of style quality, structural preservation, and multi-view consistency, providing better controllability and higher-quality style transfer for 3D artistic content creation.