基于多尺度融合与一致性优化的鲁棒多模态情感分析研究

Research on Robust Multimodal Sentiment Analysis Based on Multi-Scale Fusion and Consistency Optimization

摘要: 多模态情感分析在处理文本、音频与视觉异构模态时会产生局部特征易流失、非语言线索利用不足以及跨模态融合噪声干扰等问题，这会显著削弱模型在复杂情感场景下的判别能力与鲁棒性。针对这些问题，本文提出一种基于多尺度融合与一致性优化的鲁棒多模态情感分析模型。在特征提取阶段，模型结合DeBERTa的预训练语言模型提取文本全局语义，利用两层级联1D-CNN捕获音频与视觉局部动态特征，完成了多尺度特征提取。在跨模态融合阶段，通过联合序列拼接与动态自适应掩码，经过多层Transformer在文本语境引导下实现高信噪比的跨模态交互。在预测阶段，构建分类–回归双分支结构，通过一致性优化约束极性与强度预测的一致性，从而提升情感预测的准确性与稳定性。实验结果显示，在CMU-MOSI和CMU-MOSEI数据集上，本文方法在Acc-2、Acc-7、F1、MAE及Corr等指标均优于多种基线模型，证明了该框架具有良好的识别性能与鲁棒性。

Abstract: Multimodal sentiment analysis encounters issues such as the loss of local features, insufficient utilization of non-linguistic cues, and noise interference in cross-modal fusion when dealing with heterogeneous modalities like text, audio, and vision. These problems significantly weaken the model’s discriminative ability and robustness in complex sentiment scenarios. To address these issues, this paper proposes a robust multimodal sentiment analysis model based on multi-scale fusion and consistency optimization. In the feature extraction stage, the model combines the pre-trained language model DeBERTa to extract global semantic information from text and utilizes a two-layer cascaded 1D-CNN to capture local dynamic features from audio and vision, achieving multi-scale feature extraction. In the cross-modal fusion stage, through joint sequence concatenation and dynamic adaptive masking, high signal-to-noise ratio cross-modal interaction is achieved under the guidance of text context via multiple layers of Transformer. In the prediction stage, a classification-regression dual-branch structure is constructed, and consistency optimization is used to constrain the consistency of polarity and intensity predictions, thereby enhancing the accuracy and stability of sentiment prediction. Experimental results show that on the CMU-MOSI and CMU-MOSEI datasets, our method outperforms multiple baseline models in terms of Acc-2, Acc-7, F1, MAE, and Corr metrics, demonstrating the framework’s excellent recognition performance and robustness.