基于Hadoop的重复数据删除技术研究

Research on Duplicate Data Deletion Technology Based on Hadoop

重复数据删除技术和云计算技术正在快速发展,但是Hadoop系统也仍然存在一些影响系统归档处理效率的因素。针对这个问题,文章提出一种新的增量指纹算法和改进TTTD算法——TDOB,并将这类算法应用到Hadoop分布式系统中,分析重复数据删除技术在Hadoop中的应用。

Duplicate data deletion technology and cloud computing technology are developing rapidly,but there are still some factors that affect the efficiency of archiving processing in Hadoop system. In order to solve this problem,a new incremental fingerprint algorithm and an improved TTTD algorithm--TDOB are proposed and applied to Hadoop distributed system. The application of repeated data deletion in Hadoop is analyzed.