关联大数据管理技术:挑战、对策与实践

Big Linked Data Management: Challenges, Solutions and Practices

【目的】分析关联大数据的概念、内涵与特征,针对关联大数据管理的技术挑战,探讨关联大数据管理技术的对策和解决思路。【方法】结合NoSQL数据管理技术、分布式图计算技术、大数据流水线技术等给出应对挑战的思路,并基于此思路形成大规模图数据仓库加工系统gETL。【结果】该方法和系统在NSFC-KBMS和WDCM项目中得到了应用,实现了大规模知识型数据和生物数据的有效管理,满足了多元化的数据管理需求。【局限】需要结合应用的情况,进一步完善方法与系统。【结论】通过采用NoSQL数据存储技术、分布式图计算技术、大数据流水线技术以及gETL系统,可以很好地解决关联大数据的管理问题。

[Objective] This article analyzed the concept, connotation and characteristics of the big linked data, aiming to explore possible solutions for technical challenges facing its management. [Methods] We proposed a new model based on NoSQL data management, distributed graph computing and big data pipeline technologies, which designed and develop gETL, a large-scale graph data warehouse processing system. [Results] The proposed system was used in NSFC-KBMS and WDCM projects, which effectively manages large-scale knowledge-data and biological data. [Limitations] The proposed system could be improved with new applications. [Conclusions] The NoSQL data storage, distributed graph computing, and big data pipeline technologies, as well as the gETL system, help us address the challenges facing linked big data management.