基于注意力机制和知识蒸馏的电影评分预测

MOVIE RATING PREDICTION BASED ON ATTENTION MECHANISMAND KNOWLEDGE DISTILLATION

针对简单的因子分解机模型(Factorization Machines,FM)对于高阶交互的时间复杂度高和神经网络解决复杂问题尺寸过大问题,以电影评分预测为例,提出一种注意力机制和知识蒸馏的深度网络预测模型(Knowledge Distillation Attention Deep Network,K-ADN)。结合注意力网络区分交互特征的重要度而得到注意力值,利用深度神经网络(Deep Neural Networks,DNN)处理高阶特征组合,建立神经网络模型作为教师模型,从知识蒸馏技术出发,以教师模型确保精确度,以学生模型精简模型尺寸,以求获得更有效的评分预测结果。以豆瓣电影为数据来源进行的实验结果表明,该模型预测的精确度有所提高,通过知识蒸馏后参数量减少86%。

In view of the high time complexity of high-order interaction and the large size of complex problem solved by neural network in the simple factorization machines(FM),a knowledge distillation attention deep network(K-ADN)model is proposed based on the movie score prediction as an example.Combined with the attention network to distinguish the importance of interactive features,the attention value was obtained.Deep neural networks(DNN)were used to deal with the combination of high-order features,and the neural network model was established as the teacher model.Starting from the knowledge distillation technology,the teacher model was used to ensure the accuracy,and the student model was used to simplify the model size,so as to obtain more effective scoring prediction results.The experimental results based on Douban movie show that the accuracy of the model is improved,and the parameters are reduced by 86%after knowledge distillation.