实时面部捕捉技术在虚拟演播室中的应用研究

Real-Time Facial Capture and Reconstruction in Virtual Studio Systems

针对虚拟演播室应用中常见的人体和人脸捕捉与重建的特征识别数据量大、计算耗时长、识别准确率较低、易于导致跟踪失败等问题,提出了一种基于单摄像头的人脸识别和参数化的方法。方法基于MTCNN技术获取人脸的轮廓和特征点坐标,然后基于SFM人脸标准模型进行特征拟合,实时计算并得到摄像头画面对应的人脸模型顶点和三角面信息,从而实现数据驱动的虚拟人效果。借助这一算法,开发了虚拟演播室应用中的人体和人脸捕捉与重建原型系统。实验表明,上述算法识别特征数据快、识别准确率高、运行速度快,克服了跟踪失败问题、设备成本低,运行效果平滑稳定,用户反馈良好。

In this paper, we propose a single-camera-based face recognition and parameterization method for face capture and reconstruction in virtual studio applications, aiming at the common problems of too large recognition data, long computation time, low accuracy, and easy to lead to tracking failure.The recognition was based on MTCNN to obtain the contour and feature coordinates of human faces, and then fit features to mesh based on the SFM standard model, to calculate the face model vertex and triangle surfaces to achieve a data-driven virtual face motion and expression effect.With this algorithm, a prototype system for human and face capture and reconstruction in virtual studio applications was developed and used.The algorithm runs quickly, smoothly, and stable with low hardware cost, avoids tracking failure problems and has already been applied in actual virtual studio projects with good user feedback.