基于Scrapy框架的网络爬虫抓取实现

Implementation of Web Crawler Capture Based on Scrapy Framework

随着互联网的发展,网络数据覆盖了各个领域,但随着网络数据量的增加和数据格式的多样化,用户从海量数据中获取有价值的数据变得越来越困难。目前国内外对数据采集技术进行了研究,发现通过网络爬虫技术可以自动获取网络资源。本文以南京市二手房信息为例,设计了一个基于Scrapy框架的爬虫程序,对中西部部分地区的二手房信息进行抓取和存储,最后运用Excel数据分析,对南京市二手房资源按区域、住房类型进行分析。结果表明,该程序能够自动采集安居客户的住房信息,提高了用户获取信息的速度和质量,为用户数据分析提供了数据源。

With the development of the Internet, network data covers various fields, but with the increase of the amount of network data and the diversification of data formats, users from the massive data to obtain valuable data becomes more and more difficult. At present, data acquisition technology is studied at home and abroad, and it is found that web crawler technology can automatically acquire network resources. This paper takes the secondhand housing information of Nanjing as an example, designs a crawler program based on Scrapy framework, captures and stores the second-hand housing information of some parts of the central and western regions, and finally uses Excel data analysis to analyze the second-hand housing resources of Nanjing according to regions and housing types. The results show that the program can automatically collect the housing information of the customer, improve the speed and quality of the user to obtain information, and provide data source for user data analysis.