基于音频驱动的遮挡下人脸表情重建技术

Technology of Audio-Driven Masked Facial Expression Reconstruction

网络舆情对政治生活秩序和社会稳定的影响与日俱增,还原网络视频中刻意遮挡的人脸,有助于网络舆情管控以及掌握网络舆情攻防战的主动权。结合生成对抗网络,提出了一种基于音频驱动的遮挡下人脸表情重建方法。首先,通过音频解耦器实现音频内容与情绪分离;然后,基于双阶段遮挡下人眼关键点检测网络对头部姿态进行估计,得到头部3维姿态编码;最后,基于生成对抗网络,对生成人脸的说话口型、面部表情和头部姿态进行综合控制,实现对说话人脸的生动还原。试验结果表明,该方法可准确有效地还原人脸表情和头部姿态。

The influence of online public opinion on political life order and social stability is increasing day by day. Restoring the faces deliberately blocked in online videos is helpful to control online public opinion and seize the initiative in the battle against online public opinion. Combined with generative adversarial network, a method of audio-driven masked facial expression reconstruction is proposed. Firstly, the audio content and emotion are separated through audio decoupling device. Then, based on the two-stage masked human eye key-points detection network, the head pose is estimated to obtain the 3D head pose code. Finally, based on the generative adversarial network, the mouth shape, facial expression and head posture of the generated face are comprehensively controlled to achieve a vivid restoration of the speaker′s face. The experimental result shows that the method can effectively restore facial expressions and head poses.