基于MD5和布鲁姆过滤器的重复数据删除算法

Duplicate Data Delete Technology Based on MD5 and Bloom Filter

针对目前重复数据处理技术的低效性和不可靠性，本文提出了一种基于MD5算法和布鲁姆过滤器的重复数据删除算法。新算法采用两级布鲁姆过滤器并有效结合MDS算法的方式，在发挥布鲁姆过滤器空间效率的同时汲取了MD5算法的可靠性，使得文件级别和数据块级别的重复数据删除策略交替工作。测试分析表明，新算法性能稳定并且实现了高效且可靠的重复数据删除功能。

Aimed at the inefficiency and unreliability of the data de-duplication teclmology, a solution to this problem which based on MD5 algorithm and Bloom Filter technology is put forward in this paper.The new algorithm is the effective combination of MD5 algorithm and Bloom Filter structure.Profiting from less space occupancy of Bloom Filter and reliability of MD5 algorithm,the new algorithm can work under the alternate mode of file-level data de-duplication tactic and data-block-level data de-duplication tactic.Testing and analysis indicate that the new algorithm has achieved an efficient and secure data de-duplication function.