基于云计算技术的海量信息分布式存储研究

Research on Distributed Storage of Massive Information Based on Cloud Computing Technology

面对海量信息的有效存储,为了保证存储信息的抽取和查询的效率,研究基于云计算技术的海量信息分布式的存储方法。采用GFS作为分布式文件系统和HDFS管理节点/存储节点架构作为分布式存储技术的依据,形成极大存储容量的计算机群,对信息实行并行处理;生成事实表,分析和处理不同维度和粒度的情况下的信息后,对其实行数据聚集;采用基于云计算技术改进ETL处理算法实行海量信息抽取,存储在数据库中,用户即可根据需求实行数据库信息查询。实验结果表明,该方法的存储性能较好,物理节点的增加会提高信息的插入效率,并且抽取后的信息信噪比较高,信息查询速度较快。

In the face of the effective storage of massive information,in order to ensure the efficiency of the extraction and query of stored information,the distributed storage method of massive information based on cloud computing technology is studied.Using GFS as a distributed file system and HDFS management node/storage node architecture as the basis of distributed storage technology,a computer group is formed with a large storage capacity and implementing parallel processing of information.And the fact table is generated to analyze and process the information in different dimensions and granularity,and implement the data aggregation.The ETL processing algorithm based on cloud computing technology is improved to extract massive information and store it in the database,so that users can query the database information according to their needs.The experimental results show that the storage performance of this method is good,the increase of physical nodes will improve the insertion efficiency of information,and the SNR of the extracted information is high,and the information query speed is fast.