云存储中基于相似性的客户-服务端双端数据去重方法

Similarity-Based Client-Server Data Deduplication Method in Cloud Storage

数据去重是云存储系统提高存储率的主要手段之一,为解决查重过程中因数据量大而导致的索引I/O瓶颈和数据块指纹冲突问题,从而提高查重效率和去重准确度,提出了一种快速且安全的数据去重方法。该方法采用客户-服务器端双重检测框架,基于滑动窗口技术和Rabin指纹算法并根据文件内容确定数据块边界,摒弃传统的MD5和SHA1算法,应用第3代安全哈希函数SHA3来计算数据块的指纹。提出两级索引策略,基于数据相似性原理来加快索引查找和比对速度。试验结果表明,客户-服务器双重检测框架能有效提高系统查重效率,基于SHA3的数据块指纹计算更加精确,能有效提高去重准确度。

Data deduplication is one of the main ways to improve the storage efficiency of cloud storage system. In order to solve the problems of the I/O bottleneck of index and the fingerprint conflict of data blocks brought by large-scale data, a fast and secure data deduplication method is proposed to find duplicates more quickly and delete duplicates more accurate. In the method, a dual detection framework distributed in client and server sides is put forward, and Rabin fingerprint algorithm based on slicing window technique is adopted to determine the data block boundaries, and SHA3 is used for computing the fingerprint of the data block by abandoning the traditional MD5 and SHA1 algorithms. Furthermore, a two-level indexing strategy is proposed, which can speed up the index searching and matching speed based on data similarity. The experimental results show that this method can find duplicates efficiently by the dual detection framework, and delete duplicates accurately by data block fingerprint calculation based on SHA3.