云数据中心下重复数据删除技术研究

Research on Deduplication of Data in Cloud Data Center

云数据中心下企业数据量快速增长,使得数据中心面临严峻挑战。研究发现,存储系统中高达60%的数据是冗余的,因此云数据中心下的重复数据缩减受到越来越多的关注。以往单一存储结构模式下的存储性能评价指标(平均响应时间、磁盘I/O效率和数据冗余度),不但不能完全适应云数据这种以廉价设备为分布式存储结构的新变化,而且也难以较好地满足云服务提供商向用户做出的数据高可用性、高可靠性的SLA承诺。为此,在分析和总结云数据中心环境下数据存储的新特征之后,通过对单一存储结构下重复数据删除技术不足的剖析,提出了查询算法优化、基于SSD改进置换效率、改进的纠删码数据容错机制三条路径,以提高云数据中心下重删系统的工作效率和工作表现。最后,通过分析云服务下不同用户对IT资源需求的区别,有针对性地自动选择合适的去重时机,为从整体上改进云数据中心环境下重复删除系统操作效率指出了进一步研究的方向。

The cloud data center is facing severe challenges with the rapid growth of the data volume from enterprises.Studies have found that up to 60% of the data in storage system is redundant,so reducing the redundant data in the cloud data center is paid more and more attention.The storage performance evaluation index(average response time,disk I/O efficiency and data redundancy)in the previous single storage structure mode not only fail to adapt to the new changes of cloud data completely in the distributed storage structure with cheap devices,but also be difficult to meet SLA commitment about high availability and high reliability of the data made by the cloud service providers to users.Therefore,we propose three paths including query algorithm optimization,improved permutation efficiency based on SSD,improved erasure code data tolerance mechanism after analyzing and summarizing the new features of data storage in cloud data center and shortcoming of repeat data deletion under single storage structure,to enhance the working efficiency and performance of the system in cloud data center.Finally,by analyzing the differences between different user’s demands for IT resources in cloud services,the appropriate de-duplication timing is automatically selected in a targeted way,which points out the direction of further research for improving the efficiency for the deduplication system in cloud data center.