大模型语言学能力测评数据集构建方法研究

Research on the Construction Method and Practice of Evaluation Dataset for Linguistic Ability of Large Language Models

摘要: 本研究探索大模型语言学能力测评数据集的构建方法，提出“出题–参考答案–模型生成回复–考点生成–考点标注–基于考点评分”的标准化流程。以张谊生《现代汉语》为理论依据，借助Kimi设计两套试卷共62题，覆盖语音、句法、语义、语用维度，并选取DeepSeek、豆包、Qwen作答，由Gemini生成考点与评分规则，经人工校对形成最终数据集。研究发现：大模型在自动出题、考点生成与初步阅卷中效率较高，但精准性与规范性仍需人工干预；部分模型能自动识别题目错误，展现知识批判潜力；基于考点的结构化评分比整体打分更具可解释性。本研究为后续大模型语言学能力评测及语言学教学提供了可推广的构建方法与参考基准。

Abstract: This study explores the construction method of an evaluation dataset for the linguistic capabilities of large language models (LLMs). A standardized workflow is proposed, consisting of “question drafting - reference answer preparation - model response generation - test point generation - test point annotation - scoring based on test points.” Using Zhang Yisheng’s Modern Chinese as the theoretical foundation, two test papers comprising a total of 62 questions were designed with the assistance of the LLM Kimi. The questions cover dimensions including phonetics, syntax, semantics, and pragmatics. Three LLMs—DeepSeek, Doubao, and Qwen—were employed to generate responses, while Gemini was utilized to automatically produce test points and scoring rubrics, which were subsequently refined through manual proofreading. The findings indicate that LLMs offer high efficiency in automatic question drafting, test point generation, and preliminary scoring, yet manual intervention remains essential for ensuring precision and standardization. Notably, certain models demonstrated the ability to autonomously identify errors in the question design, revealing their potential for knowledge critique. Moreover, structured scoring based on test points proved to be more interpretable than holistic scoring. This study provides a replicable construction method and a reference benchmark for future evaluations of LLMs’ linguistic capabilities and for linguistic pedagogy.