基于时空相关度融合的语音唇动一致性检测算法

Lip Motion and Voice Consistency Algorithm Based on Fusing Spatiotemporal Correlation Degree

本文在传统发音唇动分析模型的基础上,构建一个发音唇动时空模型.提出了唇动时域特征、空域特性与语音的相关度度量方法,以及融合时空度量的语音唇动一致性检测方法.利用唇宽、唇高与音频幅度变化之间的联系获得语音唇动的时域一致性评分;通过协惯量分析法获得语音与唇部空域特征的初始相关度,并提出了针对语音、唇动自然延时的相关度修订方法;最后将时空上的得分进行融合以判断语音唇动是否一致.初步实验结果表明,对于四种不一致音视频数据,与常用的协惯量方法相比,EER(Equal Error Rate)平均下降了约8.2%.

This paper constructs a spatiotemporal lip motion model based on traditional simple pronunciation and lip motion spatial model ,and proposes methods for measuring the correlation degree between voice and the spatial ,temporal characteristic of lip motion .In addition ,a fusion scheme for the spatial and temporal correlation degree is proposed to measure the consistency of voice and lip motion .The temporal consistent score is defined as the correlation between lip shape （height and width） and the speech am-plitude .The Coinertia is used as the initial correlation degree of speech and lip spatial characteristic .Both the spatial and temporal correlation degrees are modified by audiovisual initial delay .Experimental results show that the proposed method reduces EER by about 8 .2% compared to the CoIA method .