基于大数据平台的推荐系统研究与实现

Research and Implementation of Recommendation System Based on Big Data Platform

个性化推荐系统作为人工智能一个落地场景,在社交平台、电商、生活服务等领域有着广泛的应用。为了把优选的商品提供给有需要的客户,对用户行为进行数据采集、数据清洗与存储、用户物品推荐建模、模型评估等内容进行了研究。数据采集通过客户端页面埋点技术来记录用户浏览、点击、关注等行为以及页面停留时长等数据,通过flume、kafka、hive、spark等大数据相关组件与技术完成数据采集、ETL相关操作,将用户评分表、物物余弦相似度等数据通过ALS、item-based组合召回技术,以及LR排序技术生成TOP-N推荐列表,最终经过AB测试,完成最优迭代方案版本选取。

As a landing scenario of artificial intelligence,personalized recommendation system is widely used in social platforms,E-commerce,life services and other fields.In order to provide the preferred products to the customers in need,the data collection,data cleaning and storage,user item recommendation modeling,model evaluation and other contents of user behavior are studied.Data collection records user browsing,clicking,following and other behaviors,as well as page dwell time and other data through the embedded point technology on the client page.Data collection and ETL related operations are completed through flume,kafka,hive,spark and other big data related components and technologies.Data such as user scoring table and cosine similarity of objects are generated into TOP-N recommendation list through the combined recall technology of ALS and item-based,as well as LR sorting technology,and finally tested by AB,complete the selection of the optimal iteration scheme version.