口译多模态加工中的手势认知效应研究
Investigating the Cognitive Effects of the Interpreter’s Gestures in Multi-Modal Processing
DOI: 10.12677/ML.2021.94123, PDF,    科研立项经费支持
作者: 胡敏霞*, 伍 鑫：四川大学外国语学院，四川 成都
关键词: 译员手势；认知加工；多模态分析；ELAN软件；Interpreter’s Gestures； Cognitive Processing； Multimodal Analysis； ELAN Software
摘要: 本文使用ELAN多模态话语分析软件对两段中国导演海外影片宣传的口译活动进行了分析。结果发现手势是口译加工中重要的显身特征、信息模态和认知资源，具体来说：1) 认知负荷更大时，手势数量更多，手势位置更高，手势更加快速而频繁，但主要手势类型都是节奏性手势；2) 口译认知加工中的手势、语言和副语言产品具有跨模态统一性；3) 口译质量更高的译员更倾向于模仿讲者手势，但手势类型可能发生变化。同时，手势数量也可能受到现场语境因素以及译员认知资源的影响。本研究构建的初步框架和分析工具有助于弥补口译手势多模态研究的不足，相关结论亦可促进基于询证的口译教学。
Abstract: This research uses a multi-modal discourse analysis software, ELAN, to analyse two inter-preter-mediated events featuring Chinese film directors promoting their films to an international audience. It is found that gestures represent an important embodied information modality and cognitive resource in interpreting processing. Specifically: 1) With higher cognitive load, more gestures with higher positions are made, with faster frequency and pace, although beats remain the dominant gesture type for all speakers, including interpreters; 2) Gestures align with the interpreter’s linguistic and paralinguistic products during the multimodal cognitive processing of interpreting; 3) Interpreters achieve higher interpreting quality scores through more imitation of speaker gestures, although the imitator may change the gesture type. Meanwhile, contextual factors and availability of the interpreter’s cognitive resources may also impact on the use of gestures. The preliminary framework and analysis tools built in this research could fill some of the gaps in understanding the multi-modal nature of interpreting gestures. The initial findings could also promote evidence-based interpreter education.
文章引用：胡敏霞, 伍鑫. 口译多模态加工中的手势认知效应研究[J]. 现代语言学, 2021, 9(4): 912-923. https://doi.org/10.12677/ML.2021.94123

参考文献

[1] Feyereisen, P. and De Lannoy, J. (1991) Gestures and Speech: Psychological Investigations. Cambridge University Press, Cambridge.
[2] Kendon, A. (1980) Gesticulation and Speech: Two Aspects of the Process of Utterance. In: Key, M.R., Ed., The Relationship of Verbal and Nonverbal Communication, Mouton, Hague, The Netherlands, 208-227. [Google Scholar] [CrossRef] 
[3] Bressem, J. and Ladewig, S. (2011) Rethinking Gesture Phases: Articulatory Features of Gestural Movement. Semiotica, 2011, 53-91. [Google Scholar] [CrossRef] 
[4] Kendon, A. (2004) Gesture: Visible Action as Utterance. Cambridge University Press, Cambridge. [Google Scholar] [CrossRef] 
[5] McNeill, D. (1992) Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press, Chicago.
[6] Bruner, J.S. (1990) Acts of Meaning. Harvard University Press, Cambridge.
[7] Alibali, M., Bassok, M., Solomon, K., Syc, S. and Goldin-Meadow, S. (1999) Illuminating Mental Representations through Speech and Gesture. Psychological Science, 10, 327-333. [Google Scholar] [CrossRef] 
[8] McNeill, D. (2005) Gestures and Thought. University of Chicago Press, Chicago.
[9] Goldin-Meadow, S. and Wagner, S.M. (2005) How Our Hands Help Us Learn. Trends in Cognitive Sciences, 9, 234-241. [Google Scholar] [CrossRef] [PubMed]
[10] Goldin-Meadow, S. (2011) Learning through Gesture. Wiley Interdisciplinary Reviews: Cognitive Science, 2, 595-607. [Google Scholar] [CrossRef] [PubMed]
[11] Chu, M., Meyer, A., Foulkes, L. and Kita, S. (2014) Individual Differences in Frequency and Saliency of Speech-Accompanying Gestures: The Role of Cognitive Abilities and Empathy. Journal of Experimental Psychology: General, 143, 694-709. [Google Scholar] [CrossRef] [PubMed]
[12] Gillespie, M., James, A.N., Federmeier, K.D. and Watson, D.G. (2014) Verbal Working Memory Predicts Co-speech Gesture: Evidence from Individual Differences. Cognition, 132, 174-180. [Google Scholar] [CrossRef] [PubMed]
[13] Pouw, W.T., Mavilidi, M.F., Van Gog, T. and Paas, F. (2016) Gesturing during Mental Problem Solving Reduces Eye Movements, Especially for Individuals with Lower Visual Working Memory Capacity. Cognitive Processing, 17, 269-277. [Google Scholar] [CrossRef] [PubMed]
[14] Wu, Y.C. and Coulson, S. (2014) Co-Speech Iconic Gestures and Visuo-spatial Working Memory. Acta psychologica, 153, 39-50. [Google Scholar] [CrossRef] [PubMed]
[15] Özer, D. and Göksun, T. (2020) Visual-Spatial and Verbal Abilities Differentially Affect Processing of Gestural vs. Spoken Expressions. Language, Cognition and Neuroscience, 35, 896-914. [Google Scholar] [CrossRef] 
[16] Kendon, A. (1972) Some Relationships between Body Motion and Speech. In: Seigman, A. and Pope, B., Eds., Studies in Dyadic Communication, Pergamon Press, Elmsford, 177-216. [Google Scholar] [CrossRef] 
[17] Skipper, J., Goldin-Meadow, S., Nusbaum, H. and Small, S. (2007) Speech-Associated Gestures, Broca’s Area, and the Human Mirror System. Brain and Language, 101, 260-277. [Google Scholar] [CrossRef] [PubMed]
[18] Hostetter, A.B., Alibali, M.W. and Kita, S. (2007) I See It in My Hands’ Eye: Representational Gestures Reflect Conceptual Demands. Language and Cognitive Processes, 22, 313-336. [Google Scholar] [CrossRef] 
[19] Jenkins, T., Coppola, M. and Coelho, C. (2017) Effects of Gesture Restriction on Quality of Narrative Production. Gesture, 16, 416-431. [Google Scholar] [CrossRef] 
[20] Rauscher, F.H., Krauss, R.M. and Chen, Y. (1996) Gesture, Speech, and Lexical Access: The Role of Lexical Movements in Speech Production. Psychological Science, 7, 226-231. [Google Scholar] [CrossRef] 
[21] Morsella, E. and Krauss, R.M. (2004) The Role of Gestures in Spatial Working Memory and Speech. The American Journal of Psychology, 117, 411-424. [Google Scholar] [CrossRef] [PubMed]
[22] Dargue, N., Sweller, N. and Jones, M.P. (2019) When Our Hands Help Us Understand: A Meta-analysis into the Effects of Gesture on Comprehension. Psychological Bulletin, 145, 765-784. [Google Scholar] [CrossRef] [PubMed]
[23] Goldin-Meadow, S., Cook, S.W. and Mitchell, Z.A. (2009) Gesturing Gives Children New Ideas about Math. Psychological Science, 20, 267-272. [Google Scholar] [CrossRef] [PubMed]
[24] Stieff, M., Lira, M.E. and Scopelitis, S.A. (2016) Gesture Supports Spatial Thinking in STEM. Cognition and Instruction, 34, 80-99. [Google Scholar] [CrossRef] 
[25] Chu, M. and Kita, S. (2011) The Nature of Gestures’ Beneficial Role in Spatial Problem Solving. Journal of Experimental Psychology: General, 140, 102-116. [Google Scholar] [CrossRef] [PubMed]
[26] Eielts, C., Pouw, W., Ouwehand, K., Van Gog, T., Zwaan, R.A. and Paas, F. (2020) Co-Thought Gesturing Supports More Complex Problem Solving in Subjects with Lower Visual Working-memory Capacity. Psychological Research, 84, 502-513. [Google Scholar] [CrossRef] [PubMed]
[27] Gluhareva, D and Prieto P. (2017) Training with Rhythmic Beat Gestures Benefits L2 Pronunciation in Discourse-Demanding Situations. Language Teaching Research, 21, 609-631. [Google Scholar] [CrossRef] 
[28] Novack, M. and Goldin-Meadow, S. (2015) Learning from Gesture: How Our Hands Change Our Minds. Educational Psychology Review, 27, 405-412. [Google Scholar] [CrossRef] [PubMed]
[29] Furuyama, N., Nobe, S., Someya, Y., Sekine, K. and Hayashi, S. (2005) A Study on Gestures in Simultaneous Interpreters. Interpretation Studies, 5, 111-136.
[30] Rennert, S. (2008) Visual Input in Simultaneous Interpreting. Meta, 53, 204-217. [Google Scholar] [CrossRef] 
[31] Zagar-Galvão, E. (2009) Speech and Gesture in the Booth—A Descriptive Approach to Multimodality in Simultaneous Interpreting. In: De Crom, D., Ed., Selected Papers of the CETRA Research Seminar in Translation Studies 2008, CETRA, Leuven.
[32] Zagar-Galvão, E. (2013) Hand Gestures and Speech Production in the Booth: Do Simultaneous Interpreters Imitate the Speaker. In: Carapinha, C. and Santos, I., Eds., Estudos de Linguística, Coimbra University Press, Coimbra, 115-129. [Google Scholar] [CrossRef] 
[33] Zagar-Galvão, E. (2020) Gesture Functions and Gestural Style in Simultaneous Interpreting. In: Salaets, H. and Brône, G., Eds., Linking up with Video: Perspectives on Interpreting Practice and Research, John Benjamins Publishing Company, Amsterdam/Philadelphia, 151-179. [Google Scholar] [CrossRef] 
[34] Stachowiak-Szymczak, K. (2019) Eye Movements and Gestures in Simultaneous and Consecutive Interpreting. Springer International Publishing, Cham. [Google Scholar] [CrossRef] 
[35] Gerwing, J. and Li, S. (2019) Body-Oriented Gestures as a Practitioner’s Window into Interpreted Communication. Social Science & Medicine, 233, 171-180. [Google Scholar] [CrossRef] [PubMed]
[36] Gile, D. (2009) Basic Concepts and Models for Interpreter and Translator Training. Revised Edition, John Benjamins Publishing Company, Amsterdam. [Google Scholar] [CrossRef] 
[37] Cienki, A. and Müller, C. (2008) Metaphor and Gesture. John Benjamins, Amsterdam. [Google Scholar] [CrossRef] 
[38] 齐涛云, 杨承淑. 多模态同传语料库的开发与建置——以职业译员英汉双向同传语料库为例. 中国翻译, 2020, 41(3): 126-135, 189.
[39] 王斌华. 语料库口译研究——口译产品研究方法的突破. 中国外语, 2012(3): 96-102.
[40] Goldman-Eisler, F. (1958) Speech Production and the Predictability of Words in Context. The Quarterly Journal of Experimental Psychology, 10, 96-109. [Google Scholar] [CrossRef] 
[41] Plevoets, K. and Defrancq, B. (2016) The Effect of Informational Load on Disfluencies in Interpreting. Translation and Interpreting Studies: The Journal of the American Translation and Interpreting Studies Association, 11, 202-224. [Google Scholar] [CrossRef] 
[42] Plevoets, K. and Defrancq, B. (2018) The Cognitive Load of Interpreters in the European Parliament: A Corpus-based Study of Predictors for the Disfluency uh(m). Interpreting, 20, 1-28. [Google Scholar] [CrossRef] 
[43] Xiang, X., Zheng, B. and Feng, D. (2020) Interpreting Impoliteness and Over-politeness: An Investigation into Interpreters’ Cognitive Effort, Coping Strategies and Their Effects. Journal of Pragmatics, 169, 231-244. [Google Scholar] [CrossRef] 
[44] Paas, F. and Sweller, J. (2012) An Evolutionary Upgrade of Cognitive Load Theory: Using the Human Motor System and Collaboration to Support the Learning of Complex Cognitive Tasks. Educational Psychology Review, 24, 27-45. [Google Scholar] [CrossRef] 
[45] Gile, D. (1999) Testing the Effort Models’ Tightrope Hypothesis in Simultaneous Interpreting—A Contribution. Hermes, 12, 153-172. [Google Scholar] [CrossRef] 
[46] Gile, D. (2008) Local Cognitive Load in Simultaneous Interpreting and Its Implications for Empirical Research. Forum, 6, 59-77. [Google Scholar] [CrossRef] 
[47] Xie, B. and Salvendy, G. (2000) Prediction of Mental Workload in Single and Multiple Task Environments. International Journal of Cognitive Ergonomics, 4, 213-242. [Google Scholar] [CrossRef] 
[48] Gile, D. (2011) Errors, Omissions and Infelicities in Broadcast Interpreting. Preliminary Findings from a Case Study. In: Alvstad, C., Hild, A. and Tiselius, E., Eds., Methods and Strategies of Process Research. Integrative Approaches in Translation Studies, John Benjamins, Amsterdam/Philadelphia, 201-218.