基于语义匹配的跨模态图文检索与生成研究

Research on Cross-Modal Graphic Retrieval and Generation Based on Semantic Matching

摘要: 在语义学的研究框架中，语言被视为意义的载体，而图像则是感知的表征。随着人工智能的发展，文本与图像之间的跨模态研究成为了新的热点。本文聚焦于文搜图的任务，从语义学角度分析文本如何通过模型转化为相应的图像。具体来说，本文提出了一种基于语义组合特征的CLIP + MLP模型，用于提升模型在细粒度语义对齐中的表现。此外，本文通过构造多层次文本描述，比较原始CLIP模型与CLIP + MLP模型的语义匹配能力，并采用Stable Diffusion 1.5模型进行抽象语义的图像生成测试。结果表明，CLIP + MLP模型在复杂语义结构下表现更优，而Stable Diffusion 1.5模型在抽象风格与隐喻语言的还原中亦展现出一定的语义建构能力。总而言之，本文验证了语义特征建模在跨模态任务中的关键作用，为语义驱动的图文理解提供了有力支持。

Abstract: In the research framework of semantics, language is regarded as a carrier of meaning, while images are representations of perception. With the development of artificial intelligence, cross-modal research between text and image has become a new hot spot. This paper focuses on the task of text searching for images, and analyzes how text can be transformed into corresponding images through models from a semantic perspective. Specifically, this paper proposes a CLIP + MLP model based on semantic combination features for improving the performance of the model in fine-grained semantic alignment. In addition, this paper compares the semantic matching ability of the original CLIP model and the CLIP + MLP model by constructing multi-level text descriptions, and tests the image generation of abstract semantics using the Stable Diffusion 1.5 model. The results show that the CLIP + MLP model performs better under complex semantic structures, while the Stable Diffusion 1.5 model also shows some semantic construction ability in the reduction of abstract style and metaphorical language. All in all, this paper validates the key role of semantic feature modeling in cross-modal tasks and provides strong support for semantics-driven graphic understanding.