基于语音与人脸参数化表示的跨模态稠密深度网络学习方法

Cross-modal learning based on speech and parameterized facerepresentation using densely deep networks

为了提高跨模态人脸表示与合成的性能,针对语音与人脸图像2种模态数据,提出一种基于人脸参数化表示与稠密深度网络相结合的面部生成方法。针对输入语音模态,通过对信号进行频谱变换,将一维时域信号转换到二维频率域,可提取频域上稳健的特征描述;针对输出图像模态,利用主动外观模型对不同面部区域独立建模以降低区域间的相关性,并提取紧凑的人脸参数化特征;为了获得有效的跨模态学习性能,提出采用稠密连接的深度卷积神经网络学习语音、图像2种模态的回归预测,并通过预测的人脸参数进行面部重构,所采用的深度网络模型可以加强特征传播与特征复用,有利于增强面部细节的合成。在2组音视频数据集上验证了提出方法的有效性。

To improve the performance of cross-modal face representation and synthesis,a facial synthesis method is proposed based on two modalities of speech and face image using densely deep network and cross-modal learning.First,frequency domain feature description is obtained by performing spectral transformation on the speech modal,which transforms one-dimensional time domain signal into two-dimensional frequency domain.Secondly,the active appearance model is applied to different facial regions to reduce the region correlations of the output image modal;the compact face parameterized features can then be extracted.Finally,in order to obtain effective cross-modal performance,a densely connected deep convolutional neural network is proposed to learn the regression prediction between speech and image modalities,followed by face reconstruction on the predicted parameters.The proposed deep learning model helps enhance feature communication and feature reuse,which is conducive to enhance the synthesis of facial details.Experiments on two audio and video datasets demonstrate the effectiveness of the proposed method.