基于多模态特征融合的图像编辑模型

Image Editing Model Based on Multi-Model Feature Fusion

摘要: 为了实现更可控的编辑效果，目前基于条件导向的图像编辑工作大部分都是设置了一定的文本条件引导。然而文本和图像是两种不同的模态，因此如何从不同模态的数据中有效地检索和融合不同模态的信息是图像编辑中一个很大的挑战。针对图像和文本之间的信息无法很好的深度交互的问题，提出了一种基于多模态特征融合的MFF图像编辑模型。首先利用多模态模型BLIP对编辑文本进行适应性优化，以便指导生成模型偏向生成更符合语义的图像，然后对源文本嵌入和Unet网络参数进行联合训练，再采用交叉注意力机制在U-Net网络中增强视觉感知能力来融合图文特征，最后将融合的特征经过预先训练的图文生成扩散模型得到与文本描述相关的编辑图像。在COCO数据集上的定量实验结果表明，相比于其他基线模型最好的实验结果，MFF模型在MS-SSIM度量指标提高了12.5%、在LPIPS指标上降低了26.2%，表明模型在实现图像和文本之间的特征融合方面更加有效。

Abstract: To achieve more controllable editing effects, most current condition-guided image editing works rely on text-based conditions. However, text and images represent two different modalities, making it a significant challenge to effectively retrieve and integrate information from different modalities in image editing. To address the issue of limited deep interaction between image and text information, a Multi-Modal Feature Fusion (MFF) image editing model is proposed. First, an adaptive optimization of the editing text is performed using the multi-modal model BLIP to guide the generation model towards producing images that better align with the semantics. Subsequently, joint training of the source text embeddings and U-Net network parameters is conducted, and a cross-modal attention mechanism is employed within the U-Net network to enhance visual perception capabilities for fusing visual and textual features. Finally, the fused features are passed through a pre-trained image-text generation diffusion model to obtain edited images related to the text description. Quantitative experimental results on the COCO dataset show that, compared to other baseline models, the MFF model achieved a 12.5% improvement in the MS-SSIM metric and a 26.2% reduction in the LPIPS metric, indicating its greater effectiveness in feature fusion between images and text.