基于BERT知识蒸馏的情感分析模型

BERT-Based Knowledge Distillation for Sentiment Analysis Model

摘要: 目前，BERT预训练语言模型在文本分析领域得到普遍应用，在情感分析任务上更是取得了SOTA级别的表现，但是在边缘设上部署BERT模型仍具有挑战性。而一般用于解决情感分析的传统的机器学习模型(SVM，NB和LR)较易部署，但精度不如BERT模型。本文旨在实现对两种不同方法的优势进行融合，训练出一个精度高且易部署的模型并用于解决情感分析任务。先前的工作大多是将BERT模型蒸馏进一个浅层的神经网络结构，这种方法能够减少BERT模型的参数，但依然保留了上百万的参数，难以在边缘设备上部署。本文提出将已经训练好的BERT模型设定为教师模型，将传统机器学习模型(SVM，NB和LR)设定为学生模型，并在输出层面完成知识转移。训练学生模型使用教师模输出的软标签和logits，并证明了学生模型在软标签上进行训练可以简化学生模型的学习过程，同时强调了从教师模型获得的所有文本特征之间的关系是平等的，并用蒸馏后的学生模型在IMDB数据集上进行验证。实验结果表明，蒸馏了BERT预训练语言模型知识的传统机器学习模型在测试数据上的性能得到明显提升，相比较于基线模型精度都提高了1%以上，且参数量维持在BERT模型的1/10水平。

Abstract: In recent years, pretrained language models have achieved remarkable performance in various fields, including computer vision, natural language processing, and multimodal tasks. However, while pretrained language models excel in accuracy across various tasks, they come with high computational costs and long inference times. On the other hand, traditional machine learning models are more easily deployable but often lag behind in accuracy compared to pretrained lan-guage models. This paper aims to combine the strengths of both approaches to create a highly ac-curate and deployable model. Inspired by knowledge distillation techniques (teacher-student models), this study sets a pretrained BERT language model as the teacher model and traditional ma-chine learning models as the student models. It extracts additional soft-label knowledge from the large, high-weight teacher model to train lightweight student models. The goal is to use the distilled student models to tackle sentiment analysis tasks in textual data, with a focus on the benchmark movie review dataset. The main steps include data preprocessing, feature extraction, and knowledge distillation. The research results demonstrate that traditional machine learning models, distilled with knowledge from large pretrained language models, significantly improve performance on test data, with accuracy gains of more than 1% compared to baseline models, and the number of parameters is maintained at 1/10 level of the BERT model.