海量GPR检测数据负载均衡并行处理技术

Load Balancing Parallel Processing Technology for Massive GPR Detection Data

探地雷达单机处理软件性能较低,难以满足较长里程或周期性检测的海量数据处理场景需求。针对这一问题,本文在集群模式下基于Hadoop平台的MapReduce并行计算框架,采用HDFS和MySQL的混合存储方法,对数据流进行细粒度切片,建立主从节点架构的动态调度模式,实现海量检测数据的负载均衡并行处理。在Linux系统搭建了1主+8从的Hadoop集群环境并对其进行了测试试验。结果表明:动态调度可使迭代算法达到负载均衡,高复杂度算法适用于并行处理,性能可提升100倍左右,加速比接近物理核数。

The performance of GPR single-machine processing software was low,and it was difficult to meet the needs of massive data processing scenarios with long mileage or periodic detection. To solve this problem,this paper used the hybrid storage method of HDFS and MySQL to slice the data stream in a fine-grained manner under the cluster mode based on the Hadoop platform Map Reduce parallel computing framework. The dynamic scheduling mode of master-slave node architecture was established,and the load balancing parallel processing of massive detection data was realized. A Hadoop cluster environment of 1 master with 8 slave was built on Linux system and tested. The results show that the dynamic scheduling can make the iterative algorithm achieve load balance,the high complexity algorithm is suitable for parallel processing,the performance can be improved about 100 times,and the acceleration ratio is close to the number of physical cores.