基于脑电信号和眼动信号的情感识别
Emotion Recognition Based on EEG Signals and Eye Movement Signals
DOI: 10.12677/mos.2025.141112, PDF,   
作者: 李 澳, 张恒千, 尹 钟*：上海理工大学光电信息与计算机工程学院，上海
关键词: 情感识别；对比学习；机器学习；特征融合；Emotional Recognition； Comparative Learning； Machine Learning； Feature Fusion
摘要: 本文探讨了基于脑电信号和眼动信号的多模态情感识别方法，使用了上海交通大学SEED-IV和SEED-V数据集。通过预处理脑电信号并提取功率谱密度和微分熵特征，结合眼动信号的时域和频域特征，提出了一种基于对比学习的数据融合方法，显著提高了情感识别的准确率。实验结果表明，多模态融合，尤其是脑电信号与眼动信号的结合，能显著提升识别性能；同时，微分熵特征优于功率谱密度特征，具有更好的分类准确性。
Abstract: This paper explores a multimodal emotion recognition method based on EEG and eye movement signals, using the SEED-IV and SEED-V datasets from Shanghai Jiao Tong University. After preprocessing the EEG signals, power spectral density and differential entropy features were extracted, while time-domain and frequency-domain features were extracted from the eye movement signals. A contrastive learning-based data fusion method is proposed, which effectively bridges the distribution gap between different modalities and significantly improves emotion recognition accuracy. Experimental results show that multimodal fusion, particularly the combination of EEG and eye movement signals, significantly enhances recognition performance. Additionally, the differential entropy feature of EEG signals outperforms the power spectral density feature in classification accuracy, indicating better separability in the dataset.
文章引用：李澳, 张恒千, 尹钟. 基于脑电信号和眼动信号的情感识别[J]. 建模与仿真, 2025, 14(1): 1244-1252. https://doi.org/10.12677/mos.2025.141112

参考文献

[1] Maithri, M., Raghavendra, U., Gudigar, A., Samanth, J., Prabal Datta Barua,, Murugappan, M., et al. (2022) Automated Emotion Recognition: Current Trends and Future Perspectives. Computer Methods and Programs in Biomedicine, 215, Article ID: 106646. [Google Scholar] [CrossRef] [PubMed]
[2] Jafari, M., Shoeibi, A., Khodatars, M., Bagherzadeh, S., Shalbaf, A., García, D.L., et al. (2023) Emotion Recognition in EEG Signals Using Deep Learning Methods: A Review. Computers in Biology and Medicine, 165, Article ID: 107450. [Google Scholar] [CrossRef] [PubMed]
[3] Zhang, X., Liu, J., Shen, J., Li, S., Hou, K., Hu, B., et al. (2021) Emotion Recognition from Multimodal Physiological Signals Using a Regularized Deep Fusion of Kernel Machine. IEEE Transactions on Cybernetics, 51, 4386-4399. [Google Scholar] [CrossRef] [PubMed]
[4] Wang, Q., Wang, M., Yang, Y. and Zhang, X. (2022) Multi-Modal Emotion Recognition Using EEG and Speech Signals. Computers in Biology and Medicine, 149, Article ID: 105907. [Google Scholar] [CrossRef] [PubMed]
[5] Zhu, Q., Lu, G. and Yan, J. (2020) Valence-Arousal Model Based Emotion Recognition Using EEG, Peripheral Physiological Signals and Facial Expression. Proceedings of the 4th International Conference on Machine Learning and Soft Computing, Haiphong City, 17-19 January 2020, 81-85. [Google Scholar] [CrossRef] 
[6] Zhang, J., Yin, Z., Chen, P. and Nichele, S. (2020) Emotion Recognition Using Multi-Modal Data and Machine Learning Techniques: A Tutorial and Review. Information Fusion, 59, 103-126. [Google Scholar] [CrossRef] 
[7] Zheng, W., Liu, W., Lu, Y., Lu, B. and Cichocki, A. (2019) Emotionmeter: A Multimodal Framework for Recognizing Human Emotions. IEEE Transactions on Cybernetics, 49, 1110-1122. [Google Scholar] [CrossRef] [PubMed]
[8] Liu, W., Qiu, J., Zheng, W. and Lu, B. (2022) Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition. IEEE Transactions on Cognitive and Developmental Systems, 14, 715-729. [Google Scholar] [CrossRef] 
[9] Liu, H., Zhang, J., Liu, Q. and Cao, J. (2022) Minimum Spanning Tree Based Graph Neural Network for Emotion Classification Using Eeg. Neural Networks, 145, 308-318. [Google Scholar] [CrossRef] [PubMed]
[10] Bouazizi, S., Benmohamed, E. and Ltifi, H. (2023) Enhancing EEG-Based Emotion Recognition Using PSD-Grouped Deep Echo State Network. JUCS—Journal of Universal Computer Science, 29, 1116-1138. [Google Scholar] [CrossRef] 
[11] Hwang, S., Hong, K., Son, G. and Byun, H. (2019) Learning CNN Features from DE Features for EEG-Based Emotion Recognition. Pattern Analysis and Applications, 23, 1323-1335. [Google Scholar] [CrossRef] 
[12] Li, M., Yang, B., Levy, J., Stolcke, A., Rozgic, V., Matsoukas, S., et al. (2021) Contrastive Unsupervised Learning for Speech Emotion Recognition. ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, 6-11 June 2021, 6329-6333. [Google Scholar] [CrossRef] 
[13] Pan, C., Shi, C., Mu, H., Li, J. and Gao, X. (2020) EEG-Based Emotion Recognition Using Logistic Regression with Gaussian Kernel and Laplacian Prior and Investigation of Critical Frequency Bands. Applied Sciences, 10, Article 1619. [Google Scholar] [CrossRef] 
[14] Chen, L., Su, W., Feng, Y., Wu, M., She, J. and Hirota, K. (2020) Two-Layer Fuzzy Multiple Random Forest for Speech Emotion Recognition in Human-Robot Interaction. Information Sciences, 509, 150-163. [Google Scholar] [CrossRef] 
[15] Shen, P., Changjun, Z. and Chen, X. (2011) Automatic Speech Emotion Recognition Using Support Vector Machine. Proceedings of 2011 International Conference on Electronic & Mechanical Engineering and Information Technology, Harbin, 12-14 August 2011, 621-625. [Google Scholar] [CrossRef] 
[16] Al Dujaili, M.J., Ebrahimi-Moghadam, A. and Fatlawi, A. (2021) Speech Emotion Recognition Based on SVM and KNN Classifications Fusion. International Journal of Electrical and Computer Engineering (IJECE), 11, 1259-1264. [Google Scholar] [CrossRef] 
[17] Oktavia, N.Y., Wibawa, A.D., Pane, E.S. and Purnomo, M.H. (2019) Human Emotion Classification Based on EEG Signals Using Naïve Bayes Method. 2019 International Seminar on Application for Technology of Information and Communication (iSemantic), Semarang, 21-22 September 2019, 319-324. [Google Scholar] [CrossRef] 

为你推荐