Abstract—Focused crawlers （also known as subjectoriented crawlers）, as the core part of vertical search engine, collect topic-specific web pages as many as they can to form a subject-oriented corpus for the latter ...Abstract—Focused crawlers （also known as subjectoriented crawlers）, as the core part of vertical search engine, collect topic-specific web pages as many as they can to form a subject-oriented corpus for the latter data analyzing or user querying. This paper demonstrates that the popular algorithms utilized at the process of focused web crawling, basically refer to webpage analyzing algorithms and crawling strategies （prioritize the uniform resource locator （URLs） in the queue）. Advantages and disadvantages of three crawling strategies are shown in the first experiment, which indicates that the best-first search with an appropriate heuristics is a smart choice for topic-oriented crawlingwhile the depth-first search is helpless in focused crawling. Besides, another experiment on comparison of improved ones （with a webpage analyzing algorithm added） is carried out to verify that crawling strategies alone are not quite efficient for focused crawling and in most cases their mutual efforts are taken into consideration. In light of the experiment results and recent researches, some points on the research tendency of focused crawler algorithms are suggested.展开更多
基金The National Basic Research Program of China(973 Program)(No.2010CB328104,2009CB320501)the National Natural Science Foundation of China(No.61272531,61070158,61003257,61060161,61003311,41201486)+4 种基金the National Key Technology R&D Program during the11th Five-Year Plan Period(No.2010BAI88B03)Specialized Research Fund for the Doctoral Program of Higher Education(No.20110092130002)the National Science and Technology Major Project(No.2009ZX03004-004-04)the Foundation of the Key Laboratory of Netw ork and Information Security of Jiangsu Province(No.BM2003201)the Key Laboratory of Computer Netw ork and Information Integration of the Ministry of Education of China(No.93K-9)
基金supported by the Research Fund for International Young Scientists of National Natural Science Foundation of China under Grant No.61550110248Tibet Autonomous Region Key Scientific Research Projects under Grant No.Z2014A18G2-13
文摘Abstract—Focused crawlers （also known as subjectoriented crawlers）, as the core part of vertical search engine, collect topic-specific web pages as many as they can to form a subject-oriented corpus for the latter data analyzing or user querying. This paper demonstrates that the popular algorithms utilized at the process of focused web crawling, basically refer to webpage analyzing algorithms and crawling strategies （prioritize the uniform resource locator （URLs） in the queue）. Advantages and disadvantages of three crawling strategies are shown in the first experiment, which indicates that the best-first search with an appropriate heuristics is a smart choice for topic-oriented crawlingwhile the depth-first search is helpless in focused crawling. Besides, another experiment on comparison of improved ones （with a webpage analyzing algorithm added） is carried out to verify that crawling strategies alone are not quite efficient for focused crawling and in most cases their mutual efforts are taken into consideration. In light of the experiment results and recent researches, some points on the research tendency of focused crawler algorithms are suggested.
