通过Filter抵御网页爬虫

Web Crawler by Filter resist

过滤器反爬虫的基本思想就是建立一个可以区分客户请求的总过滤器,根据用户的请求类型实现不同策略的处理。如果是一个普通的客户请求,过滤器将请求继续传递到客户请求的资源上;如果请求时一个爬虫的检索请求,就需要根据自己实际应用的需要,或者直接拒绝爬虫请求,或者延时加载爬虫的请求。另外在过滤器将所有的请求都记录到一个日志系统中,为以后网站的为系统开发﹑测试及性能调整提供依据。

The basic idea of filter anti-crawler is creating a general filter that can distinguish the different client requesl type,implementing different strategy according different client request type.If a general client request,filter forward the request to resource client＇s target Jr a crawler request,filter implements different strategy,such as refusing the re- quest or delaying load the request.furthermore,the filter can record the request to logging system.This will provide the basis for the approaching development, test or performa nce tuning.