驾驶场景下的多模态情绪识别系统设计与实现

Design and Implementation of a Multimodal Emotion Recognition System Specialized for Driving Scenarios

摘要: 在真实驾驶环境中，车内光线、道路噪声、驾驶员头部偏转等因素都会干扰情绪识别结果。单独依靠某一种数据源时，模型判断容易出现不稳定。围绕这一情况，本文设计并实现了一套驾驶场景下的多模态情绪识别系统，将视频表情、静态图像以及语音文本三类信息放入同一识别流程中。系统主要包含三个识别模块：视频流模块以ResNet-18为基础，在残差结构中加入SE注意力机制，用来加强眼部、嘴部等表情区域的响应；图像模块采用MobileNetV3，并配合旋转、裁剪、亮度扰动等数据增强方式，提高模型对不同拍摄条件的适应能力；语音文本模块分别利用Wav2Vec 2.0和BERT提取声学特征与语义特征。实验基于DMED数据集完成，融合模型准确率达到94.2%，比单模态模型提高4.1%。同时，本文使用PyQt5完成可视化界面开发，实现了数据输入、预处理、模型推理和结果展示等功能，整体上能够满足车载场景下的基本实时检测需求。

Abstract: In real driving environments, factors such as in-vehicle lighting, road noise, and driver head-pose changes can interfere with emotion recognition results. When relying on only one type of data source, the model’s judgment is prone to instability. To address this issue, this paper designs and implements a multimodal emotion recognition system for driving scenarios, integrating video-based facial expressions, static images, and speech-text information into a unified recognition process. The system mainly consists of three recognition modules: the video-stream module is based on ResNet-18 and incorporates an SE attention mechanism into the residual structure to enhance the response to facial regions such as the eyes and mouth; the image module adopts MobileNetV3 and uses data augmentation methods such as rotation, cropping, and brightness disturbance to improve the model’s adaptability to different shooting conditions; the speech-text module uses Wav2Vec 2.0 and BERT to extract acoustic and semantic features, respectively. Experiments are conducted on the DMED dataset, and the fusion model achieves an accuracy of 94.2%, which is 4.1 percentage points higher than that of the best single-modal model. In addition, this paper develops a visualization interface using PyQt5, enabling data input, preprocessing, model inference, and result display. Overall, the system can meet the basic real-time detection requirements in vehicle-mounted scenarios.