改进的数据消重方法在垂直搜索引擎中的应用

The Application of Improved Data Duplicate Elimination Method in Vertical Research Engine

本文对垂直搜索引擎中的数据消重方法做了仔细的分析,对单MD5算法和多重MD5算法在数据消重中的作用以实验作为依据,并将两者相结合得到的算法应用到电子产品垂直搜索引擎当中,取得了较好的数据消重效果,为索引模块的建立打下了基础.

This article analyzes comprehensively the data duplicate elimination method. The experiments show that the single and multiple MD5 algorithms play very important role in data duplicate elimination method. The application of the new algorithm, combined by these two algorithms, in the vertical research engine of electronic products turns out to be very effective, building up a solid founda- tion for index module.