摘要
随着业务人员对数据分析的需求不断增加,基于开源Hadoop的游戏大数据平台运行着越来越多相互依赖的作业任务。为了更加高效地完成作业管理和调度,升级了Zeus构建工作流引擎,优化Zeus任务启动过程,实现了多线程启动任务,缩短了启动时间。通过扩展Zeus系统,使其支持Oracle存储过程。同时,利用管道技术,实现了异构数据源的无缝高效共享和传输。
The big data platform for gaming service is based on Hadoop open source technology. With the demand growth of the service operation staff, there are more and more jobs which depend on each other running on the big data platform. A new working flow engine was developed on Zeus to manage and schedule jobs efficiently. In order to shorten the start time, Zeus task starting process was optimized and multi-threaded start task was achieved. Zeus was also improved to support the Oracle stored procedure. Pipeline technology was used to seamless and efficient sharing and transmission of heterogeneous data sources.
出处
《电信科学》
北大核心
2016年第2期134-139,共6页
Telecommunications Science
关键词
大数据作业
工作流
开源
big data job, work flow, open source
