基于释放和重构的科学工作流数据布局策略

A Data Placement Strategy Based on Relaxation and Reconstruction for Scientific Workflow Applications

科学工作流是典型的大数据应用,每个任务涉及的数据体量大,分布式环境下移动的网络代价高,良好的数据布局策略对于降低跨数据中心的传输代价、提高科学工作流运行效率至关重要.随着大数据时代来临,科学工作流计算任务涉及到的数据集和数据存储节点增多,使得求解数据布局最优方案变得更加困难.使用传统数据布局策略往往只能得到局部最优方案.为此,提出一种基于释放和重构的数据布局策略,使得在超大规模的解空间中尽快找到更加接近全局最优的数据布局方案.策略首先基于前期研究成果产生初始解,然后通过解的局部释放和重构,对解空间的不同邻域进行搜索,不断提高解的质量.通过实验验证并与传统方法比较,结果显示该数据布局方案数据传输代价更小,性能更好.

Scientific Workflow is the typical big data application.By defining the computing tasks and steps,it completes the massive data processing tasks for scientific computing.As the large data and the high cost of mobile networks in scientific workflow,it is essential to apply suitable data placement strategy.With the increase of the number of data centers and data sets,solving the optimal data placement strategy becomes more difficult.Traditional data placement strategies often only get local optimal solution.To solve this challenge,it presents a relaxation and reconstruction data placement strategy to quickly find the data placement scheme closer to the global optimal.Firstly,the strategy is mainly based on preliminary research results to generate the initial solution,and by the local relaxation and reconstruction,constantly improves the quality of the solution. Verified by experiments,compared with the traditional methods,the results show that the data transfer cost of our strategy is smaller.