适应大规模数据处理的动态服务私有云系统

Private cloud computing system based on dynamic service adaptable to large-scale data processing

为适应私有云环境下数据量大、计算密集、流程复杂的计算任务需求,借鉴公有云计算的相关理论与技术,结合私有云环境的特点,提出了一种适应大规模数据处理的动态服务私有云系统实现方案。该方案使用作业文件描述计算任务,以作业逻辑结构动态构建处理工作流程;通过数据流驱动服务请求,引入MapReduce并行框架进行大规模数据处理。实验结果表明:该方案能够正确有效地处理数据量大、计算密集、流程复杂的计算任务,显著提升处理效率,具有很高的实用性。

In order to deal with problem in private cloud environment caused by computing tasks with large amount of data,intensive computing and complex processing,an implementation of private cloud system based on dynamic service was proposed on the basis of public cloud computing and the characteristics of private cloud environment,which was able to adapt large-scale data processing.In this implementation,computing tasks were described by job files,processing workflows were constructed dynamically by job logic,service requests were driven by data streams and the large-scale data processing could be reflected more efficiently in MapReduce parallel framework.The experimental results show that this implementation offers a high practical value,can deal with computing tasks with large amount of data,intensive computing and complex processing correctly and efficiently.