基于任务分配和数据集副本的科学工作流数据布局

Scientific Workflow Dataset Layout Based on Task Assignment and Dataset Replicas

云环境下科学工作流的数据布局成为当前工作流研究中的一个热点问题,对科学工作流中任务和数据之间多对多关系进行分析,可以发现不同数据布局方案在数据传输上的费用各不相同,在很大程度上影响工作流的运行成本。为降低科学工作流数据集传输费用,提出一种基于任务分配和数据集副本的科学工作流数据布局方法。该方法从任务分配开始,在定量计算任务依赖度的基础上进行任务分配,根据分配结果给出一个基于数据集副本的两阶段数据布局方法,以实现科学工作流运行中传输费用的优化。实例结果表明,与工作流层方法相比,该方法可以有效降低科学工作流的运行成本。

The Data Layout(DL)for Scientific Workflow(SW)in cloud environment becomes a hot issue in current workflow research.Considering the many-to-many relationship between tasks and data in scientific workflows,it can be found that the data transmission costs of different data layout schemes are different,which can greatly affect the running cost of workflow.In order to reduce the data transmission costs in SW,this paper proposes a SW DL method based on task assignment and dataset replicas.The method starts with task assignment,assigning these tasks based on quantitative calculation of task dependencies,and then proposes a two-stage DL method based on the dataset replicas according to the assignment result,so as to achieve the optimization of transmission costs in running scientific workflows.Sample results show that this method can effectively reduce the running cost of scientific workflows compared with the workflow layer method.