面向电子商务知识服务的大语言模型评价指标研究

Research on Evaluation Metrics of Large Language Models for E-Commerce Knowledge Services

摘要: 大语言模型在电商客服场景中展现出强大的语义理解与文本生成能力，为自动化客服提供了新的技术路径。然而，现有研究主要依赖文本相似度指标评估模型性能，这类指标难以有效反映电商客服回答中的业务规则正确性与知识完整性。针对上述问题，本文提出两种面向电商知识问答任务的专业评价指标：知识要素准确率与知识要素覆盖率。知识要素准确率用于衡量模型回答中知识要素的正确程度，知识要素覆盖率用于评估模型回答对关键业务知识的覆盖情况。通过对多个主流大语言模型在电商客服问答任务上的实验分析，结果表明与传统文本评价指标相比，本文提出的指标能够更加准确地反映模型在电商知识理解与业务规则表达方面的能力，为电商平台智能客服系统的评估与优化提供科学依据。

Abstract: Large language models have demonstrated strong semantic understanding and text generation capabilities in e-commerce customer service scenarios, offering a new technical path for automated customer service. However, existing research mainly relies on text similarity metrics such as BLEU and ROUGE to evaluate model performance, which are inadequate in effectively reflecting the correctness of business rules and the completeness of knowledge in e-commerce customer service responses. To address these issues, this paper proposes two specialized evaluation metrics for e-commerce knowledge question-answering tasks: Knowledge Component Accuracy (KCA) and Knowledge Component Recall (KCR). KCA measures the accuracy of knowledge components in model responses, while KCR assesses the coverage of key business knowledge in model responses. Through experimental analysis of multiple mainstream large language models on e-commerce customer service question-answering tasks, the results show that the metrics proposed in this paper can more accurately reflect the models’ capabilities in e-commerce knowledge understanding and business rule expression compared to traditional text evaluation metrics, providing a scientific basis for the evaluation and optimization of intelligent customer service systems in e-commerce platforms.