摘要
提出一种基于Hadoop软件框架进行海量测试数据处理的解决方案。在深入研究Hadoop分布式系统构架、HDFS分布式文件系统以及Map Reduce分布式编程模型的基础上,设计并实现了二进制测试数据文件到HDFS的传输机制以及基于Map Reduce的测试数据分布式格式转换系统。最后搭建实验环境,验证了整个系统的正确性并对分布式格式转换系统进行性能评估。与本地单机相比,系统在处理海量数据时具有更高的效率及更好的可拓展性。
This paper proposes a Hadoop-based software framework for massive testing data processing scheme. Based on the related technology of Hadoop distributed system architecture, Hadoop distributed file system and MapReduce programming model, it designed and implemented the binary test data files to HDFS transport mechanism and the test data format conversion system based on MapReduce. Finally, it set up the experimental environment to verify the correctness of the whole system and did the distributed format conversion system performance evaluation. Compared with one node conversion in local, this system has higher efficiency and better expansibility in dealing with a huge amounts of data.
出处
《电子技术应用》
北大核心
2015年第7期140-143,共4页
Application of Electronic Technique
基金
陕西省自然科学基金(2014JM8311)
