大数据网络并行计算环境中生理数据流动态负载均衡

Dynamic load balancing of physiological data flow in big data network parallel computing environment

针对医疗大数据服务系统中生理数据流动态负载不均衡问题,传统方法处理能力只局限在某算子所处节点可处理的窗口范围,在数据逐渐增加的状态下处理能力不足,容易出现数据流拥塞的情况,而且忽略了对整个体系的负载分布和动态负载均衡中迁移决策的研究。为此,提出了一种新的大数据网络并行计算环境的生理数据流动态负载均衡方法。首先利用元组key的Hash值得到节点相应数据块,利用数据块记录获取相应目标节点,将数据元组输出。同时,对并行计算熵进行扩展,将其定义至异构集群,对其进行求解。将网络并行计算环境下并行计算熵看作医疗大数据服务系统中生理数据流动态负载均衡度的衡量指标,通过并行计算熵对是否需要进行负载迁移进行判断,并且通过并行计算熵确定迁移任务的方式及迁移量,从而制定迁移决策,实现大数据网络并行环境中生理数据流动态负载均衡处理。经实验验证,本文方法可行性高,计算性能及动态负载均衡性好。

In the medical big data service system,there exists the problem of the unbalanced dynamic load of physiological data flow.The processing power of the traditional method is limited to the window range that can be processed by the node where the operator is located.In the state where the data is gradually increased,the processing capacity is insufficient,the data flow congestion is easy to occur,and the load distribution of the whole system is neglected.To solve these problems,a new dynamic load balancing method for physiological data flow based on network parallel computing environment is proposed in this paper.Firstly,the Hash value of the tuple key is used to obtain the corresponding data block of the node,and the corresponding target node is obtained by using the data block record,and the data tuple is output.At the same time,the entropy of parallel computing is extended to define the heterogeneous cluster and solve it.Then,the parallel computing entropy in the network parallel computing environment is regarded as the measurement index of the dynamic load balancing of physiological data flow in the medical big data service system.Finally,by judging whether load migration is necessary by parallel computing entropy,the way and amount of migration tasks are determined by parallel computing entropy,so we can make migration decision and realize dynamic load balancing of physiological data flow in parallel environment of large data network.The experimental results show that the proposed method is highly feasible,and the calculation performance and dynamic load balance are good.