Efficient Computation Offloading of IoT-Based Workflows Using Discrete Teaching Learning-Based Optimization

As the Internet of Things(IoT)and mobile devices have rapidly proliferated,their computationally intensive applications have developed into complex,concurrent IoT-based workflows involving multiple interdependent tasks.By exploiting its low latency and high bandwidth,mobile edge computing(MEC)has emerged to achieve the high-performance computation offloading of these applications to satisfy the quality-of-service requirements of workflows and devices.In this study,we propose an offloading strategy for IoT-based workflows in a high-performance MEC environment.The proposed task-based offloading strategy consists of an optimization problem that includes task dependency,communication costs,workflow constraints,device energy consumption,and the heterogeneous characteristics of the edge environment.In addition,the optimal placement of workflow tasks is optimized using a discrete teaching learning-based optimization(DTLBO)metaheuristic.Extensive experimental evaluations demonstrate that the proposed offloading strategy is effective at minimizing the energy consumption of mobile devices and reducing the execution times of workflows compared to offloading strategies using different metaheuristics,including particle swarm optimization and ant colony optimization.

Multi-objective workflow scheduling in cloud system based on cooperative multi-swarm optimization algorithm

In order to improve the performance of multi-objective workflow scheduling in cloud system, a multi-swarm multiobjective optimization algorithm(MSMOOA) is proposed to satisfy multiple conflicting objectives. Inspired by division of the same species into multiple swarms for different objectives and information sharing among these swarms in nature, each physical machine in the data center is considered a swarm and employs improved multi-objective particle swarm optimization to find out non-dominated solutions with one objective in MSMOOA. The particles in each swarm are divided into two classes and adopt different strategies to evolve cooperatively. One class of particles can communicate with several swarms simultaneously to promote the information sharing among swarms and the other class of particles can only exchange information with the particles located in the same swarm. Furthermore, in order to avoid the influence by the elastic available resources, a manager server is adopted in the cloud data center to collect the available resources for scheduling. The quality of the proposed method with other related approaches is evaluated by using hybrid and parallel workflow applications. The experiment results highlight the better performance of the MSMOOA than that of compared algorithms.

ETL执行过程的优化研究

提出了一个ETL(Extraction-Transformation-Loading)优化框架并对ETL过程的逻辑优化进行了研究,把优化问题建模成状态空间搜索问题。每个ETL工作流看作一种状态,通过一系列正确的状态变换来构造状态空间,并且提出算法来获得最小执行时间的ETL工作流。理论分析和实践表明其具有良好效果。

基于状态空间搜索的ETL过程优化

针对Simitsis[1,2]等人提出的ETL过程优化算法中存在不足之处,提出了改进的启发式搜索算法,实验证明改进后的算法较好地降低了实际执行的代价,解决了原算法的短视性。

支持探索式研判分析的动态ETL流程服务

ETL(Extract-Transformation-Loading)作为一种将不同来源的原始数据转换为有效信息的数据集成手段,可为基于综合信息的研判分析提供支持.然而,在当前信息共享和大数据等发展趋势下,各行业研判分析活动表现出多样化、动态性的特征,往往需要面对大量动态、突发的数据抽取、转换及加载需求,传统预定义式的ETL方式难以快速实现.为此,提出一种支持探索式研判分析的动态ETL流程服务,包括探索式研判分析的基本思路、动态ETL流程服务的实现及ETL处理并发性能优化方法,并结合某公安联合作战指挥应用案例对动态ETL流程服务进行了验证,实际应用表明该服务能够支持以"边构造边执行"的方式实现用于研判分析的ETL流程.

面向数据仓库的ETL工具的研究与实现

阐述了ETL工具在数据仓库系统的建设过程中所处的重要地位,提出了一个基于工作流引擎的ETL系统设计。系统实现一个轻量级的工作流引擎作为ETL流程的调度中心,利用元数据保存记录ETL流程的信息,创建基于数据库引擎的ETL操作组件库来完成从数据源到数据目标的数据映射,并初步实现了对大数据环境数据抽取的支持,从而构建了一个具有良好可扩展性和开放性的ETL系统。

一种基于改进的链式MapReduce的并行ETL应用

介绍了并行ETL的相关工作和常见的处理多MapReduce作业流程的方法；提出一种改进的链式MapReduce框架．并将此框架应用于一个并行ETL工具，同时提出一些针对ETL处理的流程级优化规则，使ETL流程产生更少的MapReduce作业，从而减少I／O以及网络传输的消耗；利用某省份手机上网数据与Hive进行了大数据对比实验，结果表明．本ETL工具的性能平均比Hive快10％～20％。

ETL集群优化技术研究与实现

介绍了ETL功能及其处理技术,探讨了库表及文件抽取方式,数据转换技术策略以及数据加载技术,设计了SQL脚本自动解析获取元数据的实现方法,研究了集群优化技术在ETL数据处理中的应用。

粒子群算法在分布式ETL任务调度中的应用

随着分布式数据环境越来越复杂,ETL工具要面临数据源多、分布地域广和海量数据等因素带来的挑战。原有的集中式ETL工作流优化理论不能满足现在复杂数据环境的要求。介绍了如何将基于置换的离散型粒子群算法应用到分布式ETL任务优化调度问题上,主要工作围绕ETL工作调度模型、算法编码设计、目标函数选择等内容来展开,给出了分布式ETL工作调度策略的实现过程和伪代码。理论分析和实验证明了实际应用的有效可行性。

An Effective Cloud Workflow Scheduling Approach Combining PSO and Idle Time Slot-Aware Rules

Workflow scheduling is a key issue and remains a challenging problem in cloud computing.Faced with the large number of virtual machine(VM)types offered by cloud providers,cloud users need to choose the most appropriate VM type for each task.Multiple task scheduling sequences exist in a workflow application.Different task scheduling sequences have a significant impact on the scheduling performance.It is not easy to determine the most appropriate set of VM types for tasks and the best task scheduling sequence.Besides,the idle time slots on VM instances should be used fully to increase resources'utilization and save the execution cost of a workflow.This paper considers these three aspects simultaneously and proposes a cloud workflow scheduling approach which combines particle swarm optimization(PSO)and idle time slot-aware rules,to minimize the execution cost of a workflow application under a deadline constraint.A new particle encoding is devised to represent the VM type required by each task and the scheduling sequence of tasks.An idle time slot-aware decoding procedure is proposed to decode a particle into a scheduling solution.To handle tasks'invalid priorities caused by the randomness of PSO,a repair method is used to repair those priorities to produce valid task scheduling sequences.The proposed approach is compared with state-of-the-art cloud workflow scheduling algorithms.Experiments show that the proposed approach outperforms the comparative algorithms in terms of both of the execution cost and the success rate in meeting the deadline.

一种基于Storm平台的ETL方案实现

随着互联网在各个领域的不断发展,数据开始呈现结构多样化与体积海量化。面对海量数据的冲击,如何提高ETL的效率至关重要。针对“信息孤岛”中数据来源及格式皆不统一、数据采集实时性差的问题,提出垂直切分ETL工作流和水平切分待处理数据集,建立一种基于Storm平台的流式ETL处理方案。同时,针对Storm在进行任务分配时对工作节点CPU负载不敏感的缺点,通过定时任务记录工作节点的CPU负载信息,对Storm调度器的slot分配方式进行优化,使得Storm集群的负载更加均衡。实验结果证明该方案可有效提高ETL的处理效率,同时针对slot分配优化可有效地提高系统稳定性与处理效率。

基于状态空间搜索的ETL执行优化

数据抽取-转换-重载(ETL)是构建和维护数据仓库的基本构件,由于它处理的是海量数据,如何加快响应时间成为值得研究的问题。对ETL过程的逻辑优化进行了研究,把优化问题建模成状态空间搜索问题。把每个ETL工作流看作一种状态,通过一系列正确的状态变换来构造状态空间,并且提出算法来获得最小执行时间的ETL工作流。

一种针对云计算的ETL流程创建及优化方法

该文针对现有ETL工作流程的创建和执行过程中的不足，提出了一种基于逻辑模型的ETL工作流程优化方法，并针对云计算模式对ETL工作流程的执行进行进一步优化，通过测试这种两阶段优化方法可以大大提高ETL工作流程的执行效率。

基于多Agent与工作流的分布式ETL引擎的研究

针对传统ETL工具集中式执行方式的不足,提出了一种基于多Agent与工作流相结合的分布式ETL引擎的体系结构。该体系结构由一个主控引擎和多个执行引擎组成,执行引擎可自主地向主控引擎注册执行服务,并利用分布式计算和多线程并行计算技术,实现由多个执行引擎协同执行ETL工作流,从而提高整个系统的灵活性和吞吐率。实验结果表明,该引擎具有较好的可扩展性和负载平衡性能,并提高了执行效率。

ETL工作流活动优先级的确定及并行实现

ETL流程是一个以数据为中心的工作流,对ETL工作流的执行过程进行论述,提出了一个算法,计算ETL工作流中各个活动的执行优先级,在工作流执行中为优先级相同且相互之间没有依赖关系的活动集创建多个线程,通过并行执行这些活动,提高了ETL工作流的执行效率。实验结果表明,所提出的并行算法与串行算法比较,在数据量足够大的情况下,加速比可接近理想值,加速比随着数据量增大而提高。

Monte Carlo Simulation-Based Robust Workflow Scheduling for Spot Instances in Cloud Environments

When deploying workflows in cloud environments,the use of Spot Instances(SIs)is intriguing as they are much cheaper than on-demand ones.However,Sls are volatile and may be revoked at any time,which results in a more challenging scheduling problem involving execution interruption and hence hinders the successful handling of conventional cloud workflow scheduling techniques.Although some scheduling methods for Sls have been proposed,most of them are no more applicable to the latest Sls,as they have evolved by eliminating bidding and simplifying the pricing model.This study focuses on how to minimize the execution cost with a deadline constraint when deploying a workflow on volatile Sls in cloud environments.Based on Monte Carlo simulation and list scheduling,a stochastic scheduling method called MCLS is devised to optimize a utility function introduced for this problem.With the Monte Carlo simulation framework,MCLS employs sampled task execution time to build solutions via deadline distribution and list scheduling,and then returns the most robust solution from all the candidates with a specific evaluation mechanism and selection criteria.Experimental results show that the performance of MCLS is more competitive comparedwithtraditionalalgorithms.

混合云环境面向安全科学工作流数据布局策略

为解决混合云环境下科学工作流数据布局问题,在考虑数据的安全需求的前提下,以优化跨数据中心传输时延为目标,提出了一种混合云环境下面向安全的科学工作流布局策略。分析数据集的安全需求以及数据中心所能提供的安全服务,提出安全等级分级规则;设计并提出基于遗传算法和模拟退火算法的自适应粒子群优化算法(adaptive particle swarm optimization algorithm based on SA and GA,SAGA-PSO),避免算法陷入局部极值,有效提高种群多样性;与其它经典布局算法对比,基于SAGA-PSO的数据布局策略在满足数据安全需求的同时能够大大降低传输时延。

基于遗传算法的建筑布局中工作流的构建与优化研究——以沈阳市沈河区硫酸厂旧址更新设计为例

遗传算法是一种模拟自然进化的计算模型,在数学和计算机科学等多个领域有着广泛的应用,在建筑设计中同样具有重要的应用价值。文章提出了一种基于遗传算法的逆向优化方法,并构建了设计工作流,为前期批量生成合理的建筑布局方案提供了一套解决方法。此工作流通过参数化形态建模和遗传算法对建筑性能进行优化分析,并以增加建筑冬至日采光时间作为优化目的,在冬至日平均采光时间由最小的0.59 h提高到1.12 h,结果表明该工作流可用于批量生成优化布局方案,为建筑布局优化提供了一种有效的解决思路。

Cloud control for IIoT in a cloud-edge environment

The industrial Internet of Things(IIoT)is a new indus-trial idea that combines the latest information and communica-tion technologies with the industrial economy.In this paper,a cloud control structure is designed for IIoT in cloud-edge envi-ronment with three modes of 5G.For 5G based IIoT,the time sensitive network(TSN)service is introduced in transmission network.A 5G logical TSN bridge is designed to transport TSN streams over 5G framework to achieve end-to-end configuration.For a transmission control protocol(TCP)model with nonlinear disturbance,time delay and uncertainties,a robust adaptive fuzzy sliding mode controller(AFSMC)is given with control rule parameters.IIoT workflows are made up of a series of subtasks that are linked by the dependencies between sensor datasets and task flows.IIoT workflow scheduling is a non-deterministic polynomial(NP)-hard problem in cloud-edge environment.An adaptive and non-local-convergent particle swarm optimization(ANCPSO)is designed with nonlinear inertia weight to avoid falling into local optimum,which can reduce the makespan and cost dramatically.Simulation and experiments demonstrate that ANCPSO has better performances than other classical algo-rithms.

基于异构消息匹配QoS全局最优E-workflow组合研究

提出了一种分布消息匹配的工作流组合方法。移动agent代理E-service,构成一个自治的感知单元,移动agent通过语义的协商确认匹配关系,E-workflow组合可以通过移动agent的演化完成。通过接口消息对应匹配的方法,完成了服务之间关系的确定,建立工作流组合的全局结构,将E-workflow的QoS全局优化转化为一个多目标的优化问题,通过基于快速排序算法改进的NSGA-Ⅱ算法进行求解。通过仿真表明方法实现了分布自组织和全局适应性的功能。