CMPTA：预训练大模型在多模态情感分析任务中的应用研究

CMPTA: Exploring the Application of Pre-Trained Large Language Models in Multimodal Sentiment Analysis

摘要: 大语言模型(LLMs)在自然语言处理领域取得了显著进展，但将其有效迁移至多模态情感分析(MSA)任务仍面临巨大挑战。主要难点在于如何弥合异构模态(如视觉、音频)特征与预训练文本大模型语义空间之间的鸿沟。现有方法多依赖复杂的深度融合网络或昂贵的全量微调，难以充分利用大模型的推理与泛化能力。为此，本文提出了一种轻量级的跨模态伪Token适配器(Cross-Modal Pseudo-Token Adapter, CMPTA)。该方法并不破坏大模型的原有参数，而是通过高效的注意力机制，将非文本模态特征转化为LLM可理解的“伪Token”(Pseudo-Tokens)，并以软提示(Soft Prompts)的形式注入文本输入序列，从而实现多模态信息与文本语义的深度对齐。此外，本文还系统探究了伪Token数量对模型语义对齐效果的影响规律。实验结果表明，CMPTA能够有效激发大模型的多模态情感理解能力，其性能优于当前的先进基线方法，验证了该框架的有效性与泛化能力。

Abstract: Large Language Models (LLMs) have achieved remarkable progress in Natural Language Processing, yet effectively adapting them to Multimodal Sentiment Analysis (MSA) tasks remains a significant challenge. The core difficulty lies in bridging the gap between heterogeneous modal features (e.g., visual, acoustic) and the semantic space of pre-trained text models. Existing approaches often rely on complex deep fusion networks or expensive full fine-tuning, failing to fully leverage the reasoning and generalization capabilities of LLMs. To address this, we propose a lightweight Cross-Modal Pseudo-Token Adapter (CMPTA). Instead of disrupting the original parameters of the LLM, this method employs an efficient attention mechanism to transform non-textual modal features into “Pseudo-Tokens” understandable by the LLM. These tokens are then injected into the text input sequence as Soft Prompts, achieving deep alignment between multimodal information and textual semantics. Furthermore, we systematically investigate the impact of the number of pseudo-tokens on semantic alignment. Experimental results demonstrate that CMPTA effectively stimulates the multimodal sentiment understanding capability of LLMs, outperforming state-of-the-art baselines, thereby validating the effectiveness and generalization ability of the framework.