基于Python的社交网站用户行为数据采集方法

A Python-based method for collecting user behavior data on social media sites

传统数据采集方法存在适用范围较窄、重复性工作量较大等问题,导致社交网站用户行为数据采集效率较差,提出基于Python的社交网站用户行为数据采集方法。采用情境标记法确定社交网站用户行为数据采集时机,基于Python语言搭建一个以MJU采样算法为URL地址管理中心的Scrapy爬虫框架,执行Scrapy爬虫框架完成社交网站用户行为数据的采集流程。实验结果表明,本文方法在采集社交网站的用户行为数据时,采集速率为830个/h,验证了该方法具有快速性。

Traditional data collection methods have problems such as narrow applicability and large repetitive workload,resulting in poor efficiency in collecting user behavior data on social media sites.Therefore,a Python-based method for collecting user behavior data on social media sites is proposed.Using situational tagging to determine the timing of collecting user behavior data on social media sites,a Scrapy crawler framework based on Python language with MJU sampling algorithm as the URL address management center is constructed,and the Scrapy crawler framework is executed to complete the process of collecting user behavior data on social media sites.The experimental results show that the method proposed in this paper has a collection rate of 830 user behavior data per hour when collecting user behavior data from social media sites,which verifies the speed of the method.