云环境下应用感知的动态重复数据删除机制

A Dynamic Deduplication Method with Application-Aware in Cloud Environment

针对传统在线/离线重删对云存储系统中重删效率不高的问题,采用混合重复数据删除(Hy-Dedup)机制,通过融合在线和离线两种方式进行有效的数据重删。该方案在线重删阶段根据负载类型对指纹索引进行聚类分组,设置不同重删阈值来评估数据流的空间局部一致性,提高了缓存命中率;离线重删阶段采用延迟敏感的方法,对在线阶段缓存没有命中的重复块进行精确重删。通过这种混合方式在保持系统的I/O性能和吞吐量的前提下,显著减少了写入云存储的重复数据量。实验结果表明,与iDedup机制相比,Hy-Dedup机制可将在线重删率提高35.9%,磁盘空间需求减少41.36%,并且能够在云存储系统中实现高准确率的重删,提升重删效率,节省存储空间。

A hybrid deduplication method(Hy-Dedup)is adopted to solve the problem that the deduplication efficiency in the cloud storage system is not high for traditional mode online/offline deduplication,and the method performs effective data deduplication by combining online and offline modes.This method clusters fingerprint indices according the type of loads in online deduplication stage by adopting the fingerprint caching technology.The temporal local consistency of the duplicated data in data stream is estimated and the spatial local consistency is evaluated by setting different deduplication thresholds to reduce the disk fragments.The problem that the cache cannot be hit because lack of local consistency in the offline deduplication phase will be solved.The duplicated data is significantly reduced by this method while maintaining the I/O performance and the system throughput.Experimental results and a comparison with iDedup show that Hy-Dedup improves the online deduplication ratio by up to 35.9%and the disk capacity requirement reduces by 41.36%.It is concluded that the proposed method can achieve high-deciding deduplication in the cloud storage system,improve deduplication efficiency,and save storage space.