大数据计算与存储分离技术实验分析

Experimental Analysis of the Separation Technology of Big Data Computing and Storage

随着大数据的发展,传统的集群混合部署模式逐渐暴露出管理难、资源竞争、性能相互制约等问题。计算与存储分离部署方案不断发展,开始呈现其优势,但是在解决计算与存储资源弹性扩展、高容错和高可靠问题上,且具备兼顾虚拟机和容器两种部署方式的能力要求上,目前的方案都相形见绌。基于此,提出并实现了一种基于Ceph的大数据平台分离存储方案,解耦了计算与存储之间的绑定关系,同时实现了资源的弹性利用,并兼容了虚拟机和容器的两种大数据部署方式。最后我们搭建了试验环境对方案的功能及性能进行了测试,对上述能力进行了验证。

With the development of big data,the traditional mixed deployment mode of cluster gradually exposed the problems of management difficulty,resource competition,performance constraints and so on.The separation of computing and storage deployment scheme has been developing continuously,and has begun to show its advantages.However,in solving the problems of elastic expansion,high error tolerance and high reliability of computing and storage resources,and having the ability to take both virtual machine and container into account,the current schemes are dwarfed.Based on this,this paper proposes and implements a seperated storage scheme of big data platform based on Ceph,which decouples the binding relationship between computing and storage,realizes the elastic utilization of resources,and is compatible with two big data deployment modes of virtual machine and container.Finally,we set up a test environment to test the function and performance of the scheme,and verify the above capabilities.