国产化环境下的海量小文件数据分布式存储技术

Distributed Storage Technology of Massive Small FileData in Localization Environment

为缓解单一存储设备存储海量小文件的压力,提出了一种国产化环境下的海量小文件数据分布式存储技术。利用聚类算法实现海量小文件合并。以达到最大均衡度为目标,在多项约束条件下利用人工鱼群算法求解分布式存储方案。按照分布式存储方案将海量小文件数据迁移到存储节点及其存储设备上,完成海量小文件数据分布式存储。结果表明:14个存储节点和28个存储设备的内存占用较为均衡,内存资源利用率较高。将小文件样本迁移并存储到节点的过程中,分布式存储均衡度整体波动均超过设定的阈值1.0,说明分布式存储均衡度较好,证明了所提存储技术的有效性。

In order to alleviate the pressure of a single storage device to store large amounts of small files,a distributed storage technology for large amounts of small file data in a domestic environment is proposed.Using clustering algorithm to merge large amount of small files.Taking the maximum degree of equilibrium as the goal,the artificial fish swarm algorithm is used to solve the distributed storage scheme under multiple constraints.According to the distributed storage scheme,the massive small file data is migrated to the storage nodes and their storage devices to complete the distributed storage of massive small file data.The results show that the memory occupation of 14 storage nodes and 28 storage devices is relatively balanced,and the utilization rate of memory resources is high.In the process of migrating and storing small file samples to nodes,the overall fluctuation of distributed storage balance exceeds the set threshold of 1.0,indicating that the distributed storage balance is good,which proves the effectiveness of the proposed storage technology.