大数据时代下爬虫技术应用与研究——以标讯快车项目为例

随着互联网快速发展和大数据时代的来临,Web数据逐渐庞大,如何有效并快速地从互联网上获取到用户自身需要的信息是亟需解决的问题,网络爬虫技术应运而生,它是搜索引擎抓取系统的重要组成部分。文章是以标讯快车项目为研究目标,依托本学院在大数据方面的研究优势,结合该院IT特色,具有较强的实际意义和社会意义。

With the rapid development of the Internet and the advent of big data era, it is urgent to solve the problem of howto get the information needed by users from the Internet effectively and quickly. Network crawler technology emerges as the times require, it is an important part of search engine grab system. This paper is based on the standard express project as the research goal,relying on the research advantage of big data in this college, combined with the IT characteristics of the institute, has a strong practical and social significance.