融合多尺度结构与通道注意力机制的多模态情感识别研究

Multimodal Emotion Recognition Based on Multi-Scale Structures and Channel Attention Mechanism

摘要: 基于现有生理信号情感识别研究中存在的特征表达能力不足、模态干扰严重及情感分析精度较低等问题。对此，本文提出一种基于脑电图(EEG)、心电图(ECG)与皮肤电(EDA)信号融合的多模态情感识别方法。针对生理信号特征表达能力弱的问题，设计了融合标准卷积与1D-Inception结构的多尺度特征提取模块(MSI-Block)，在提取丰富特征的同时控制参数复杂度；引入通道交互注意力机制提升关键模态响应，减少冗余干扰；并采用双向长短期记忆网络(BiLSTM)建模融合特征的时序信息。实验在DEAP数据集上验证了该方法的有效性，在Valence、Arousal和Valence-Arousal四分类任务中准确率分别达到90.72%、89.48%和83.62%，显著优于传统单模态与双模态方法，表明所提方法具有良好的情感识别性能与稳定性。

Abstract: To address the limitations in existing physiological signal-based emotion recognition studies—such as insufficient feature representation, severe modality interference, and low emotion classification accuracy—this paper proposes a multimodal emotion recognition method that fuses electroencephalogram (EEG), electrocardiogram (ECG), and electrodermal activity (EDA) signals. To enhance the feature extraction capability for physiological signals, a Multi-Scale Inception Block (MSI-Block) is designed by integrating standard convolution with a 1D-Inception structure, enabling rich feature extraction while controlling model complexity. A channel-wise interactive attention mechanism is introduced to enhance the response of key modalities and suppress redundant interference. Furthermore, a bidirectional long short-term memory (BiLSTM) network is employed to model the temporal dependencies of the fused features. Experiments conducted on the DEAP dataset demonstrate the effectiveness of the proposed approach, achieving classification accuracies of 90.72%, 89.48%, and 83.62% in valence, arousal, and valence-arousal tasks, respectively. The results significantly outperform traditional unimodal and bimodal approaches, indicating that the proposed method provides robust and reliable emotion recognition performance.