基于自适应序列帧长度的端到端式唇语识别算法

An End-to-End Lip-Reading Recognition Algorithm Based on the Adaptive Length of Frame Sequence

唇语识别的提出为信息安全、辅助驾驶等多个新兴领域提供了崭新的思路,但现有唇语识别算法架构复杂、收敛速度慢,难以载入便携式设备以实现端到端的唇语识别。基于卷积神经网络(CNN)和双向长短期记忆(BLSTM)循环神经网络,本文提出了一种自适应序列长度的端到端式唇语识别神经网络算法。首先,该算法通过Dlib特征点定位法确定视频流中特征区域的位置;然后将位置信息传入CNN神经网络中进行预处理并得到相应的开关信号;最后,将开关信号传入BLSTM中控制其帧序列的长度。该算法对帧间底层的时间信息建模更加充分并能载入到端到端便携式设备上。经实验验证,该算法在数据集MIRACL-VC1上的有效精度达67.2%,与最先进的自适应序列唇语识别算法相比提升了11.2%。

The proposed lip-reading recognition provides a brand new idea for many emerging fields such as information security and assisted driving.Existing lip-reading recognition algorithms feature complex schemes,slow convergence rates,and difficulty to load into portable devices for end-to-end application.Based on convolutional neural network(CNN)and bidirectional long short-term memory(BLSTM)recurrent neural network,this work reports on an end-to-end neural network model for lip-reading recognition with adaptive sequence length.The algorithm assigns the location of feature regions in the video stream by Dlib feature point localization method and transfers them into the CNN neural network for pre-processing to obtain the switch signal,and then transfers the obtained switch signal into the BLSTM to control the length of frame sequence.The algorithm models the underlying temporal information between frames more fully and can be loaded into the end-to-end portable devices.The algorithm has been experimentally validated to achieve a recognition accuracy of 67.2%on the dataset MIRACL-VC1,offering an absolute improvement of 11.2%to the previous state-of-the-art adaptive lip-reading recognition algorithm.