英语写作智能评估工具的比较研究——以批改网与DeepSeek为例

A Comparative Study of Automatic English Writing Assessment Tools—A Case Study of “Pigai” and “DeepSeek”

摘要: 本研究旨在比较传统自动写作评估工具“批改网”与新兴大语言模型“DeepSeek”在评估英语专业学生作文档次时的评分效度。本研究直面一线外语教学中写作评估“高需求–低反馈”的核心痛点，以113篇英语专业二年级学生的作文为样本，以TEM4分项评分标准为效标，采用相关分析与典型案例分析法，对比了批改网与DeepSeek在作文评分中的效度。研究发现：(1) DeepSeek总分与教师总分呈强正相关(r = 0.890, p < 0.01)，而批改网仅为中等相关(r = 0.429, p < 0.01)；(2) DeepSeek总分与教师内容分高度相关(r = 0.827, p < 0.01)，批改网与教师内容分相关性较弱(r = 0.292, p < 0.01)，与教师语言分为中等相关(r = 0.552, p < 0.01)；(3) 典型案例分析进一步揭示，批改网无法识别跑题作文等根本性内容问题，其反馈集中于词汇、句法等表层语言特征，而DeepSeek在各分数段上均与教师判断更为一致，其反馈能够触及论证逻辑、结构层次等内容层面。上述发现表明，DeepSeek在写作评估的整体效度及对内容维度的把握上均显著优于批改网，为教师在技术选择和教学设计上提供了清晰的决策依据。

Abstract: This study aims to compare the scoring validity of the traditional automated writing assessment tool “Pigai” and the emerging large language model “DeepSeek” in evaluating the quality of English major students’ compositions. Addressing the core challenge of “high demand, low feedback” in frontline foreign language writing instruction, this study uses 113 essays written by second-year English majors as the sample, adopts the TEM4 analytic rating scale as the criterion, and employs correlation analysis and typical case analysis to compare the validity of Pigai and DeepSeek in essay scoring. The findings reveal that: (1) DeepSeek scores show a strong positive correlation with teacher scores (r = 0.890, p < 0.01), while Pigai shows only a moderate correlation (r = 0.429, p < 0.01); (2) DeepSeek scores are highly correlated with content scores given by teacher (r = 0.827, p < 0.01), whereas Pigai shows a weak correlation with content scores given by teacher (r = 0.292, p < 0.01) and a moderate correlation with language scores given by teacher (r = 0.552, p < 0.01); (3) typical case analysis further reveals that Pigai fails to identify fundamental content issues such as off-topic essays, with its feedback primarily focusing on surface-level language features like vocabulary and syntax, while DeepSeek demonstrates greater consistency with teacher judgments across all score levels, with its feedback addressing content dimensions such as argumentation logic and structural organization. These findings indicate that DeepSeek significantly outperforms Pigai in both overall scoring validity and the assessment of content quality, providing a clear empirical basis for teachers in technology selection and instructional design.