基于深度学习的Spark电影推荐系统设计

Movie Recommendation System Based On Din And Spark

传统的推荐算法,如协同过滤等只能进行输入特征之间的简单组合,不能很好的挖掘特征之间的隐含信息,表达能力不强,很难为用户提供个性化推荐,近些年来深度学习在推荐系统领域的应用取得了很好的推荐效果。本文主要采用DIN深度学习模型作为排序层算法,采用Embedding技术作为快速召回算法,并利用TensorFlow Server建立模型服务;采用HDFS,Spark,Kafka,Flink等大数据存储,传输,计算框架完成特征的存储、离线计算与实时计算,通过对用户历史行为以及实时特征的采集处理,结合推荐算法完成对用户的离线推荐与实时推荐,生成用户感兴趣的Top-N电影列表,通过SSM框架实现完整的推荐系统前后端搭建。该系统保证了运行时的稳定性,推荐实时性,并在一定程度上提升了推荐效果。

Traditional recommendation algorithms,such as collaborative filtering,can only perform simple combinations of input features,so it is difficult to mine the hidden information between features,therefore cannot provide users with personalized recommendations.In recent years,the application of deep learning in the field of recommendation systems has achieved very good personalized recommendation effects.This article mainly uses the DIN deep learning model as the sorting layer algorithm,the Embedding technology as the recall layer algorithm,and establishes a model service through TensorFlow Server.The system uses HDFS,Spark,Kafka,Flink and other big data frameworks to complete feature storage,offline computing,and real-time computing.Using big data technology to collect and process users'historical behaviors and real-time features,the recommendation algorithm can complete offline and real-time recommendations for users to generate a list of Top-N movies that users are interested in.The system implements a complete front-end and back-end construction of the recommendation system through the SSM framework.The combination of various technologies ensures the stability of the runtime,the real-time performance of the recommendation,and improves the recommendation effect to a certain extent.