基于聚类优化与跨模态协同的多模态情感识别

Clustering-Based Optimization and Cross-Modal Collaborative Learning for Multimodal Sentiment Analysis

摘要: 多模态情感识别旨在通过融合文本、视频和音频等多种模态信息实现更准确的情感理解。现有方法在捕获跨模态互补信息时存在冗余问题，且难以建立有效的跨模态情感关联。研究提出了一种分层协同学习框架，通过特征聚类优化与跨模态协同学习机制解决上述问题。该方法首先采用聚类算法对多模态特征进行分组优化，结合注意力权重分配机制降低冗余并突出显著特征；随后设计跨模态协同学习模块，利用交叉注意力机制实现文本引导的初步学习以及音频与视频模态的相互引导学习，从而增强多模态表示能力。在MOSI和MOSEI两个公开数据集上的实验结果表明，所提方法在多个指标上取得具有竞争力或领先的性能，验证了该方法在提升多模态情感识别性能方面的有效性。

Abstract: Multimodal emotion recognition aims to achieve more accurate emotion recognition by integrating information from multiple modalities, such as text, video, and audio. However, existing methods often suffer from feature redundancy when capturing cross-modal complementary information and struggle to establish effective cross-modal emotional correlations. To address these challenges, we propose a Hierarchical Collaborative Learning framework that combines clustering optimization with cross-modal collaborative learning. Specifically, a clustering algorithm is first applied to optimize the grouping of multimodal features, together with an attention-based weight allocation mechanism, to reduce redundant information and emphasize salient features. Subsequently, a cross-modal collaborative learning module is designed to perform text-guided initial learning and mutual guidance learning between audio and visual modalities through a cross-attention mechanism, thereby enhancing multimodal representation ability. Experimental results on the public MOSI and MOSEI datasets demonstrate that the proposed method achieves competitive performance across multiple evaluation metrics, validating the effectiveness of the proposed method in improving multimodal emotion recognition performance.