汉语口语对话中姿态与语音信息关系初探

Relationships between gestures and speech in spontaneous Chinese speech

信息交互方式多种多样,以语音和姿态的表达最为自然,因此提高人机交互能力就需了解交际过程中的这2种模态对信息表达之间的关系。该文介绍了语音与姿态关系的相关理论和产生模型,并以电视访谈节目中自然对话的视频和音频数据为研究对象,对汉语普通话语音和姿态信息在交际过程中的关系进行了初步的研究。在语音学和姿态标注的基础上,分析了口语对话中焦点重音与姿态动作之间的关系,以及韵律边界和姿态边界之间的关系。研究发现语音上重音表达往往伴随较强烈的手部动作,而且此时手和头部动作之间有互补的现象;韵律边界和姿态边界没有时间上的对应关系,但有很大的相关性,这些结果都支持语音与姿态表达之间的关联理论。

Although humans communicate in various ways,the most natural expressions are related to speech and gestures.This paper describes a pilot study of the relationship between the two modalities of speech and gesture for Chinese spontaneous speech to improve the interactive capability of human computer interaction systems(HCI).The paper uses a speech and gesture production model with a multimodal coding scheme to annotate four video and audio clips.The speech stress is then correlated with the hand gesture ampli...