一种基于Spark时效化协同过滤推荐算法

A Real-time Collaborative Filtering Recommendation Algorithm Based on Spark

针对传统的批量学习的基于模型的协同过滤算法对新用户(物品)更新缓慢,模型重训练成本高且扩展性不足,对噪音数据的处理有待提高,尤其是随着数据量的增长和时效性要求越来越高,挖掘其中的知识变得越来越困难等问题,对置信权重在线协同过滤算法进行改进。引入自适应软边缘,提出二阶在线优化方法处理在线协同过滤中问题的新算法(Soft Confidence Weighted Online Collaborative Filtering,SCWOCF),并在Spark流处理推荐框架下利用四组真实数据与相关算法作对比测试。实验结果表明,新算法能够及时处理用户(物品)的动态变化,并提升推荐的实时性和准确性,降低计算成本,对噪声数据健壮性更强。

Focused on some drawbacks of traditional collaborative filtering algorithms based on model of batch learning,such as updating slowly for new users or items,highly retraining cost and expanding difficultly,and handling noise data need to be improved,especially, being more and more difficult for knowledge mining with growing data and the requirement of real-time,the online collaborative filtering algorithm of confidence weighted is improved. In order to solve these problems, a new algorithm named SCWOCF （ Soft Confidence Weighted Online Collaborative Filtering） was proposed. In this algorithm,the adaptive soft margin was added and the second order online optimization methodology was used to solve online collaborative filtering problems. Finally, several experiments with four real-world datasets was conducted compared with some similar algorithms on the Spark stream processing recommendation framework. The results show that the new algorithm can timely handle dynamic change of users and items,promoting the real-time and accuracy of recommenda-tion,reducing cost of computation,increasing robustness to noise data.