一种基于云芯一号的分布式文件多维混合存储机制

A Multi-dimensional Hybrid Decision Mechanism for storage of distributed file based on Cloud-Core V1.0

近年来,硬件加速逐渐广泛用于分布式存储服务,各种存储服务的稳定性受到用户的额外关注。FPGA是最常用的硬件加速设备,用来对存储文件进行存储前的压缩和读取前的解压。长时间的压缩/解压工作,使得FPGA存在服务中断等不稳定因素。尽管存储服务中断并不像以前其他云服务中断那样具有灾难性,但它们的影响可能比以前更广泛。为了解决上述问题,文章提出了一种基于拥有自主产权的云芯一号加速卡的分布式多维混合文件存储机制。首次在多Hadoop集群中,考虑芯片温度、芯片频率、文件存储大小等三个方面的因素,结合层次分析法(AHP),进行多维度的存储节点选择。文章利用传统的软件压缩和其他FPGA加速卡作对比实验,实验结果表明本文提出硬件加速设备和所述多维文件存储机制可以有效地降低多Hadoop集群的宕机率,提高文件存储压缩率和集群中各个节点的利用率。

In recent years, hardware acceleration has been widely used in distributed storage services, and the stability of various storage services has received additional attention from users. FPGAs are the most commonly used hardware acceleration devices for pre-storage of stored files and decompression before reading. Long-term compression/decompression work causes FPGAs to have unstable factors such as service interruption. Although storage service disruptions are not as catastrophic as other cloud service outages, they may be more widespread than before. In order to solve the above problems, this paper proposes a distributed multi-dimensional hybrid file storage mechanism based on Cloud-Core V1.0 accelerator card with independent property rights. For the first time in a multi-Hadoop cluster, considering the three factors of chip temperature, chip frequency, file storage size, combined with the Analytic Hierarchy Process(AHP), multi-dimensional storage node selection. In this paper, the traditional software compression and other FPGA accelerator cards are used for comparison experiments. The experimental results show that the hardware acceleration device and the multi-dimensional file storage mechanism can effectively reduce the downtime of multi-Hadoop clusters, improve the file storage compression rate and the resource utilization of node in each cluster.