摘要
针对现有中文句子级唇语识别技术存在的视觉歧义、特征提取不充分导致识别准确率偏低的问题,提出了一种基于时域卷积网络,采用三维时空卷积的中文句子级唇语识别算法——3DT-CHLipNet(Chinese LipNet based on 3DCNN,TCN)。首先,针对特征提取不充分的问题,所提算法采用了比长短期记忆网络(LSTM)感受野更大的时域卷积网络(temporal convolutional network,TCN)来提取长时依赖信息;其次,针对中文唇语识别中存在的“同型异义”视觉歧义问题,将自注意力机制应用于中文句子级唇语识别,以更好地捕获上下文信息,提升了句子预测准确率;最后,在数据预处理方面引入了时间掩蔽数据增强策略,进一步降低了算法模型的错误率。在最大的开源汉语普通话句子级数据集CMLR上的实验测试表明,与现有中文句子级唇语识别代表性算法相比,所提算法的识别准确率提高了2.17%至23.99%。
Existing lip-reading algorithms for Chinese sentences are inadequate at feature extraction and visual ambiguity resolution,which leads to low accuracy.Aiming at this problem,this paper proposed a lip-reading algorithm for Chinese sentences based on temporal convolutional network and 3D convolutional neural network(3DT-CHLipNet).Firstly,it used a temporal convolutional network to extract long-term features from lip dynamics sequences,which has a much larger receptive field than the long short term memory network.Secondly,in order to minimize the visual ambiguity in Chinese lipreading,it adopted a Transformer model with the self-attention mechanism to capture the context information and improve the accuracy of sentence prediction.Finally,it introduced a temporal masking data augmentation strategy in the data preprocessing to further reduce the error rate of the algorithm.Comparison experiments on CMLR,the largest open-source sequence-to-sequence Chinese mandarin lip reading dataset,show that the improvement in accuracy over representative lip reading algorithms for Chinese sentences ranges from 2.17%to 23.99%.
作者
刘培培
贾静平
Liu Peipei;Jia Jingping(School of Control&Computer Engineering,North China Electric Power University,Beijing 102206,China)
出处
《计算机应用研究》
CSCD
北大核心
2023年第9期2596-2602,共7页
Application Research of Computers
基金
北京市自然科学基金资助项目(4162056)
中央高校基本科研业务费资助项目(2016MS33)。
关键词
中文唇语识别
深度学习
时域卷积网络
注意力机制
Chinese lip recognition
deep learning
temporal convolutional network
attention mechanism