面向大数据的数据管理平台架构研究与应用

Research and Application of Data Management Platform Architecture Based on Big Data

大数据管理对传统的基于关系数据库管理系统和文件管理服务器的处理能力和数据吞吐量提出了挑战,为了更好的利用大数据改进企业统计的科学性和及时性,实现应用系统获取结构化和非结构化数据,更快、更精准的统计,研究开发支持结构化和非结构化数据统一管理的数据管理平台成为一项非常迫切并且重要的任务。针对于大数据的数据管理平台具备存储容量大、存储多种媒体形式的特点,设计了基于Spark平台的并行计算数据管理平台原型S-BDMP和基于S-BDMP的数据服务管理平台的应用方案。大数据管理平台原型S—BDMP可以广泛应用于海量的结构化数据分析和非结构化数据的存储和处理。S-BDMP通过在国网甘肃省电力公司信息通信公司信息系统中的集成测试,使得原有信息系统在海量数据处理中有了很大的性能提升,并很大程度提高了系统吞吐能力。

Big data management challenges the traditional based on relational database management system and document management server processing power and data throughput,in order to better use big data improved enterprise statistical science and timeliness,the application system to achieve access to structured and unstructured data,faster and more accurate statistics and research and development support both structured and unstructured data unified management platform for data management has become a very urgent and important task.For in large data platform for data management should have characteristics of large storage capacity,storage of a variety of media forms,we design the spark platform based parallel computing for the application of the S- BDMP data service management platform based data management platform prototype S- BDMP.Large data management platform prototype S- BDMP can be widely used in massive amounts of structured data analysis and unstructured data storage and processing.S- BDMP passed in the national network of Gansu Province electric power company information and communication company information system integration testing,the original information system has been much improvement in performance and greatly improve the system throughput capacity in dealing with massive data.