基于Redis的分布式搜索引擎研究

Research on Distributed Search Engine Based on Redis

针对互联网网内信息搜索效率低下问题,设计了以Redis数据库以及Map-reduce思想为核心的分布式搜索引擎框架。为了应对互联网信息时效性强、更新快、难以被准确检索的特点,基于该框架设计了分布式爬虫、分布式索引建立、分布式链接分析算法。该框架明显提高了信息处理的效率,为分布式搜索引擎的搭建提供有效模板。经过测试,与以基于其它主流框架搭建分布式搜索引擎相比,基于Redis的分布式搜索引擎在爬虫爬取、索引生成、链接分析性能方面均有提升。

To tackle the inefficiency of searching information through the Internet, a distributed search engine based on the Redis Data Base and map-reduce pattern was devised. To better adapt to the situation of the Internet at present, which is characterized by time-sensitive, fast-update and searching time-consuming features, three techniques including distributed crawler, distributed index construction and distributed link analysis algorithm is applied within our distributed search engine. The framework greatly elevate the efficiency of the information processing and provide an effective template for the construction of the distributed search engine. After testing, compared with the search engines based on the other prevalent frameworks, the performances of three as- pects including crawling, index generation and link analysis of the distributed search engine based on the Redis Data Base all have a obvious elevation.