CEENet：结合对比学习与视觉嵌入的情绪识别方法

CEENet: Emotion Recognition with Contrastive Learning and Visual Embeddings

摘要: 针对当前上下文情绪识别研究中抽象情绪语义学习不足、上下文线索融合方式单一以及面部表情特征分离质量不佳等困难，文章提出了一种结合对比学习与视觉嵌入增强的情绪识别网络CEENet (Context Embedding Emotion Network)。该网络通过并行提取背景、身体、面部以及基于视觉–语言模型生成的视觉嵌入共四路特征作为情绪识别的上下文，进一步加深了对高级情绪语义的挖掘。为实现多来源上下文特征的高效融合，文章设计了多流特征注意力模块，利用自适应注意力机制动态调整各上下文信息在情绪识别中的权重，强化情绪相关线索并抑制噪声。此外，针对面部表情特征的分离，文章提出了一种多尺寸特征增强的面部表情嵌入网络，并引入一种创新的三层级数据增强三元表情对比学习方案，显著提升了表情嵌入的判别力。实验结果表明，CEENet在Emotic和CAER-S等上下文情绪识别基准测试上分别达到了34.15%的平均精确率均值和88.25%的分类准确率，同时本研究中的面部表情嵌入网络也在FEC基准测试中取得了86.1%的准确率，均优于同类型主流方法。最后，消融实验进一步验证了主干特征提取结构、特征融合结构及多层级数据增强策略在提升模型性能方面的有效性。

Abstract: To address the challenges in current context-aware emotion recognition research, including inadequate learning of abstract emotional semantics, limited strategies for fusing contextual cues, and unsatisfactory disentanglement of facial expression features, this paper proposes CEENet (Context Embedding Emotion Network), an emotion recognition network that integrates contrastive learning with visual embedding enhancement. CEENet extracts four parallel streams of features as contextual cues for emotion recognition, namely background, body, face, and visual embeddings generated by a vision-language model, thereby enhancing the modeling of high-level emotional semantics. To achieve efficient fusion of multi-source contextual features, a Multi-Stream Feature Attention module is designed to dynamically adjust the weights of different contextual information through an adaptive attention mechanism, thereby strengthening emotion-relevant cues while suppressing noise. In addition, to address the disentanglement of facial expression features, this paper proposes a facial expression embedding network with multi-scale feature enhancement and introduces an innovative three-level data-augmentation-based triplet expression contrastive learning scheme. The proposed method significantly improves the discriminability of expression embeddings. Experimental results show that CEENet achieves a mean average precision of 34.15% and a classification accuracy of 88.25% on context-aware emotion recognition benchmarks such as Emotic and CAER-S, respectively. Meanwhile, the facial expression embedding network developed in this study attains an accuracy of 86.1% on the FEC benchmark, outperforming existing mainstream methods. Finally, ablation studies further verify the effectiveness of the backbone network design, the fusion network design, and the multi-level contrastive learning strategy in improving emotion recognition performance.