物体指称行为中涉手模式与指称语的互动研究

A Study of the Interaction Between Hand Gestures and Referring Expressions in Object-referring Practice

模式间的互动是多模态研究领域的重要课题之一。基于现有涉手模式和指称语的研究,本文以指称物体为切入点,系统地探讨两种模式的互动方式。首先,本文自上而下地提出了切分多模态指称行为语料的四层单位:指称情境、指称活动、指称任务和指称手段。然后,本文结合自建语料库中的实例从形式互动和角色互动两个方面探讨涉手模式和指称语的互动方式。两种模式的形式互动包括四种方式:(1)单个涉手模式与单个指称语互动;(2)单个涉手模式与两个或多个指称语互动;(3)两个或多个涉手模式与单个指称语互动;(4)两个或多个涉手模式与两个或多个指称语互动。角色互动包含三种方式:(1)涉手模式为"主",指称语为"宾";(2)涉手模式为"宾",指称语为"主";(3)涉手模式和指称语处于平等地位。

The interaction between modes is an important research topic of muhimodality studies. This study intends to systematically investigate the patterns of interaction between hand gestures and referring expressions utilized in object- referring practice. As the first step of investigation, four units for segmenting the multimodal data of referring practice are proposed, namely, referential context, referential activity, referential task and referential means. Then, with instances from the self-built corpus, this study analyzes the patterns of interaction between the two modes in terms of form interaction and role interaction. The form interaction between the two modes include four patterns ： （ 1 ） the interaction between one hand gesture and one referring expression ; （2） the interaction between one hand gesture and two or more referring expressions ; （ 3 ） the interaction between two or more hand gestures and one referring expression; and （4） the interaction between two or more hand gestures and two or more referring expressions. The role interaction between the two modes include three patterns： （1） hand gesture in primary position while referring expression in secondary position; （2） hand gesture in secondary position while referring expression in primary position; （3） hand gesture and referring expression in equal positions.