体裁与语向差异对人工与大型语言模型笔译评分行为的影响

The Impact of Genre and Translation Direction on Human and Large Language Model Scoring Behaviors in Translation Assessment

摘要: 本文基于多面Rasch模型(Many-Facet Rasch Model, MFRM)，探讨体裁与语向差异对人工评分员与大型语言模型(LLMs)在笔译评分中的影响。研究以北京市某211高校商务英语专业30名本科生完成的八项平行翻译任务为语料，涵盖科技、商务、新闻与议论文四类文本的汉译英与英译汉方向。根据权威测评报告选取四个主流大语言模型(ChatGPT-4o、DeepSeek、通义Qwen-2.5与腾讯元宝)，并与两位专家评分员共同参与评分。结果表明，人工评分整体偏宽，而大型语言模型普遍偏严；体裁差异显著影响评分严厉度，反映出评分者对文本功能与语言密度的敏感性。语向效应分析显示，人工评分在英译汉方向更为严格，腾讯元宝在汉译英方向偏严，而ChatGPT-4o与通义Qwen-2.5在双向评分中保持较高一致性。研究结果表明，大语言模型已具备初步的体裁识别与语向适配能力，但仍存在一定的严厉度偏移。本文为智能笔译测评系统的校准机制、体裁化教学反馈及跨语向评分公平性优化提供了实证参考。

Abstract: Drawing on the Many-Facet Rasch Model (MFRM), this study investigates how genre and translation direction shape the rating behavior of human raters and large language models (LLMs) in translation assessment. The dataset comprises eight parallel translation tasks completed by 30 Business English undergraduates at a 211 university in Beijing, covering scientific, business, news, and argumentative texts in both Chinese-English and English-Chinese directions. Four mainstream LLMs (ChatGPT-4o, DeepSeek, Tongyi Qwen-2.5, and Tencent Yuanbao) were selected based on authoritative evaluation reports and, together with two expert human raters, evaluated the translations using a unified scoring rubric. The findings show that human raters were generally more lenient, whereas all LLMs exhibited a consistent tendency toward stricter scoring. Genre exerted a significant influence on rating severity, indicating raters’ sensitivity to textual function and information density. With respect to translation direction, human raters were stricter in the English-Chinese tasks, while Tencent Yuanbao demonstrated higher severity in the Chinese-English direction. In contrast, ChatGPT-4o and Tongyi Qwen-2.5 maintained relatively high consistency across both directions. Overall, the results suggest that LLMs have begun to develop initial capacities for genre recognition and direction adaptation, although noticeable severity biases remain. These findings offer empirical support for the calibration of intelligent translation assessment systems and provide pedagogical implications for genre-based instructional feedback and the enhancement of cross-directional fairness in translation evaluation.