生成式人工智能与教师英语写作任务评改一致度研究

Consistency between GAI’s and Teachers’ Rating and Corrective Feedback in English Writing Task

摘要: 本研究旨在探讨生成式人工智能与教师在评改英语写作任务时的一致程度。共15名大学生参加了为期三个月的雅思写作任务2教学实验，由两位教师和两种生成式人工智能(KIMI与ChatGPT 4.0)分别对120份作文进行独立评分并提供文字版评改建议。通过计算生成式人工智能与教师评分的一致性，发现两种生成式人工智能评分的内部一致性高于两位教师的内部一致性；人工智能与教师间的一致程度一般。从成绩分布情况看，生成式人工智能成绩分布呈现比较集中的趋势，且成绩评定偏高于学生实际英语写作能力。通过编码分析教师与生成式人工智能对学生英语作文的评改建议，发现两种生成式人工智能在对120份作文的评改建议中稳定且一致地遵守了雅思作文2的评改维度；而两位教师则有时会有主观倾向，评改建议不能覆盖雅思作文2的四个评分维度。研究还发现，教师与生成式人工智能在评改建议提出方式方面也有差别。研究结论表明目前教师尚不能完全依赖人工智能评改作文，但可与人工智能协作开展写作任务评改，提高效率的同时，有效避免因教师人为因素产生的不一致性。

Abstract: This study aims to explore the consistency between generative artificial intelligence (GAI) and teachers in assessing English writing tasks. A total of 15 university students participated in a three-month teaching experiment focused on IELTS Writing Task 2. Two teachers and two generative AI systems (KIMI and ChatGPT 4.0) independently scored 120 essays and provided written feedback. By calculating the consistency between the AI and teacher scores, it was found that the internal consistency of the two AI systems was higher than that of the two teachers, while the consistency between the AI and teachers was moderate. In terms of score distribution, the AI systems exhibited a relatively concentrated trend, with scores generally higher than the students’ actual English writing proficiency. Through coding and analysis of the feedback provided by teachers and AI on the students’ essays, it was observed that the two AI systems consistently adhered to the assessment criteria of IELTS Writing Task 2 across all 120 essays. In contrast, the teachers occasionally displayed subjective tendencies and did not fully cover all four scoring dimensions of the task. Additionally, differences were noted in the manner in which teachers and AI delivered their feedback. The study concludes that, at present, teachers cannot fully rely on AI for essay assessment. However, collaboration between teachers and AI in evaluating writing tasks can enhance efficiency while effectively mitigating inconsistencies and subjectivity arising from human factors.