计算密集型大流量数据的接力计算与动态分流处理

Relay computation and dynamic diversion of computing-intensive large flow data

针对当前大流量数据计算速度慢、服务器端计算压力大等问题,提出一套计算密集型大流量数据的接力计算与动态分流处理模型。首先,在分布式环境下,使用内存型数据存储技术确定计算任务的运算量与复杂等级,同时利用节点资源能力对节点进行排序;然后,动态分配任务到不同节点进行并行计算,并采用一种接力处理模式完成计算任务的分解,以有效保证高流量复杂运算任务的性能和精度要求。通过分析对比,可知在万级以上数据量的情况下,多个节点比单个节点的运行时间更短、计算速度更快;而且,将该模型应用于实际时,发现它不仅能在高并发场景下减少运行时间,而且也能节省更多计算资源。

In view of the problems such as the slow computation of large flow data,the high computation pressure on the server,a set of relay computation and dynamic diversion model of computing-intensive large flow data was proposed.Firstly,in the distributed environment,the in-memory data storage technology was used to determine the computation amounts and complexity levels of the computation tasks.At the same time,the nodes were sorted by the node resource capacity,and the tasks were dynamically allocated to different nodes for parallel computing.Meanwhile,the computation tasks were decomposed by a relay processing mode,so as to guarantee the performance and accuracy requirements of high flow complex computing tasks.Through analysis and comparison,it can be seen that the running time of multiple nodes is shorter than that of the single node,and the computation speed of multiple nodes is faster than that of the single node when dealing with data volume of more than 10000 levels.At the same time,when the model is applied in practice,it can be seen that the model can not only reduce the running time in high concurrency scenarios but also save more computing resources.