摘要
大数据背景下的海量分布式数据流处理技术要求较高的实时性和成功率。为了提高分布式环境下数据流实时处理的实时性,采用Hadoop分布式系统平台,提出了一种Map Reduce工作流实时调度优化方法。该方法采用2个相关参数来动态调整每个作业在队列中的重要程度,以便对工作流的优化做出动态的判断。此外,为了减少工作流整体的执行时间,对关键作业能否本地化进行判断和时间对比,从而尽量增加map任务的本地化。实验结果显示,提出的工作流实时调度优化方法在响应时间和调度成功率方面表现出较好的性能。
Under the background of big data,massively distributed data stream processing technology requires high real-time and success rate.In order to improve the real-time performance of real-time processing of data streams in distributed environment,a real-time scheduling optimization method of Map Reduce workflow based on Hadoop distributed system platform was proposed.In addition,in order to reduce the overall execution time of the workflow,the proposed strategy judge whether the key jobs can be localized and compare with time,so as to maximize the localization of map tasks.The experimental results show that the proposed workflow real-time scheduling optimization method shows better performance in terms of response time and scheduling success rate.
作者
梁肖
孙丽
Xiao LIANG;Li SUN(Department of Computer Science,Hebei Professional College of Political Science and Low,Shijiazhuang 050061,China)
出处
《机床与液压》
北大核心
2019年第6期150-155,共6页
Machine Tool & Hydraulics
基金
Research topic of Hebei Department of human resources and social security in 2016-Research on the transfer of employment channel based on large data in rural labor force(JRS-2014-1106)
Research project of Hebei Professional College of Political Science and Low in 2016 "Research on the problems and Countermeasures of students’Entrepreneurship in E-commerce Major"(ZF20160613)~~
