SCE大数据平台接入的研究与实现

Research and Implementation of Big Data Platform Access of SCE

SCE是中国科学院计算机网络信息中心自主研发的用于聚合各超算中心、高校、研究所高性能计算机资源的科学计算中间件。大数据技术近年来飞速发展,SCE的用户在海量数据处理与大数据计算上的需求日益提升。为了解决这个问题,我们希望可以实验性的将大数据平台与SCE实现对接,为用户提供大数据平台服务。本文利用自建的Hadoop平台为例,并编写程序实现了将其接入SCE中。实验证明,可以通过SCE向Hadoop平台进行提交、查询等一系列操作,正常的将Hadoop平台接入。

SCE is a scientific computing middleware independently developed by the Computer Network Information Center(CNIC)of the Chinese Academy of Sciences for the aggregation of high-performance computer resources in various super-computing centers,universities and research institutes.Big data technology has developed rapidly in recent years,and users of SCE are increasingly demanding massive data processing and big data computing.To solve this problem,we hope to experimentally connect the big data platform with SCE to provide users with big data platform services.This article takes the self-built Hadoop platform as an example,and writes a program to implement it into the SCE.The experiment proves that a series ofoperations such as submitting and querying to the Hadoop platform can be performed through the SCE,and the Hadoop platform is normally accessed.