Web渗透测试的信息抓取策略研究

Research on Information Fetching Strategy of Web Penetration Test

文章研究了在渗透测试中Web站点的信息抓取的问题.针对Web渗透测试对于信息抓取的全面性、高效性的需求,本文通过对Web站点进行研究分析,提出了基于导航链接的网络爬虫策略,并通过减少迭代次数对传统的MD5去重算法进行了改进,优化了URL去重的效率.通过实验验证表明,该爬虫策略的信息抓取覆盖率和网页下载效率均有所提高.

This paper investigates the problem of information crawling in Web site in penetration testing. In order to meet the requirement of high efficiency and comprehensiveness of information fetching in Web penetration test, in this paper, we researched and analyzed the Web site, and proposed a web crawler strategy based on navigation link. Besides, to optimize the efficiency of URL de-emphasis, we improved the traditional MD5 de-emphasis algorithm by reducing the number of iterations. The experimental results show that the coverage of information fetching and web page download efficiency are raised with the Strategy of Web crawler.