基于交叉注意力融合与模糊C均值聚类的多模态抑郁识别

The Multimodal Depression Recognition Based on Cross-Attention Fusion and Fuzzy C-Means Clustering

摘要: 抑郁症的早期识别对有效干预至关重要，捕获深层次的音频特征和具有长距离依赖关系的文本特征并进行特征融合提升模态耦合能力是当前主要挑战。基于深度学习和多模态数据建立端到端的抑郁症识别模型：采用VGGish-NetVLAD-GRU模型提取音频深层时序特征；RoBERTa-BiLSTM模型捕捉文本长程语义依赖；通过交叉注意力融合实现语音–文本特征的动态权重分配与跨模态语义对齐；引入模糊C均值聚类算法(Fuzzy C-means, FCM)，基于概率隶属度对情感相近的样本进行软划分，实现抑郁症分类。实验结果表明，在EATD-Corpus和CMDC中文数据集上该模型准确率分别达97.0%和94.0%，F1值分别达97.0%和94.0%。在EATD-Corpus数据集上设计对照实验，交叉注意力缺失准确率下降13%，FCM缺失准确率降低7%。

Abstract: Early identification of depression is crucial for effective intervention, with current primary challenges being the extraction of deep audio features and long-range dependent text features, along with enhancing modal coupling capability through feature fusion. An end-to-end depression recognition model was established based on deep learning and multimodal data: the VGGish-NetVLAD-GRU model was employed to extract deep temporal audio features; the RoB-ERTa-BiLSTM model captured long-range semantic dependencies in text; dynamic weight allocation and cross-modal semantic alignment of speech-text features were achieved through parametric cross-attention fusion; the Fuzzy C-means (FCM) clustering algorithm was introduced to perform soft partitioning of samples with similar emotional characteristics based on probabilistic membership, thereby enabling depression classification. Experimental results demonstrated that the model achieved accuracies of 97.0% and 94.0% on the EATD-Corpus and CMDC Chinese datasets, respectively, with corresponding F1 scores of 97.0% and 94.0%. Ablation studies on the EATD-Corpus dataset showed that the absence of cross-attention led to a 13% decrease in accuracy, while the absence of FCM resulted in a 7% reduction in accuracy.