基于后缀结构进行数据块优化的重复数据删除系统

De-Duplication System Based on Suffix Structure for the Block Size Optimization

为进一步提高重复数据删除系统的性能,提出基于数据分块的后缀数组SA和最长公共前缀LCP进行数据块优化的重复数据删除系统。系统首先将输入的数据流进行第一次分块,识别出相同的分块并给分块编号,创建分块编号序列的SA和LCP表,识别出最大重复队列和非重复数据块,进一步得出优化的超级块大小,然后以超级块为单元进行第二次数据分块并保存数据压缩结果。实验表明,相比于固定分块,该系统能实现给定输入流较好的压缩性和数据重构性。

To further improve the performance of data de-duplication system,the paper brings out a de-duplication system based on data blocks’ suffix array and longest common prefix for the block size optimization.The method first chunks input data into segments with a first size,then identifies the segments and create the indexes queue for the identifiers,and then create the suffix array and the longest common prefix structure from the indexes,next determines a second optimize size based on repeated indexes sequences and non-repeated indexes,finally chunks the input data into segments based on super chunks and saves the compression result.The result shows that it achieves a better compression ratio and object reconstruction for the given input data comparing the original fixed chunk segment size.