大数据DAG任务流调度平台技术研究与应用

Research and application of big data DAG task flow scheduling platform technology

文章提出了一种大数据DAG任务流调度平台技术,其能够基于DAG任务流进行调度,以及对大数据处理流程进行调度。为了实现这一目的,文章从几个方面进行详细设计,即架构设计、协议设计、引擎设计、引擎热加载机制、DAG结构、资源介质机制、调度算法、回调机制、信号机制。使用基于拖拽的方式进行流程配置,降低用户的使用难度,最终实现在企业实时/离线大数据处理流程中承担所有任务调度工作。

The article proposes a big data DAG task flow scheduling platform technology,which can schedule based on DAG task flow and schedule big data processing processes.In order to achieve this goal,the article conducts detailed design from several aspects,namely architecture design,protocol design,engine design,engine hot loading mechanism,DAG structure,resource medium mechanism,scheduling algorithm,callback mechanism,and signaling mechanism.Using a drag and drop based approach for process configuration,reducing the difficulty of user usage,ultimately achieving all task scheduling tasks in the real-time/offline big data processing process of the enterprise.