基于分布式的算法封装技术研究与应用

Research and Application of Distributed Algorithm Encapsulation Technology

随着大数据时代的到来,电力行业相关数据的存储总量已达一定规模,若能从中挖掘有关知识,进而指导业务转型,将有助于行业快速发展。机器学习和统计知识则是将这些数据转换为知识的关键,但现有机器学习算法独立分散且用法相对复杂,同时模型也无法复用,这在一定程度上造成了资源的冗余和浪费。为支撑电力行业的各类数据分析项目,建设快速灵活、简便易用、分布式以及可复用的企业算法中台,需要综合考虑算法封装方法及分布式架构的设计与实现。因此针对电力行业特点及实际需要,详细研究了基于分布式的算法封装及算法优化技术。所作出的主要贡献有以下两点,一是提出了一种可扩展的分布式算法封装框架,二是基于分布式算法封装框架,融合一种算法优化技术,为非专业人员进行数据分析提供了便利。

With the advent of the era of big data,the total amount of relevant data storage in the power industry has reached a certain scale.If relevant knowledge can be mined from it,and then guide the business transformation,it will help the rapid development of the industry.Machine learning and statistical knowledge are the keys to turning this data into knowledge.However,the existing machine learning algorithms are independent and scattered,their usage is relatively complex,and the model cannot be reused,which leads to the redundancy and waste of resources to a certain extent.In order to support various data analysis projects in the power industry and build a fast,flexible,easy to use,distributed and reusable enterprise algorithm middle platform,it is necessary to comprehensively consider the algorithm encapsulation method and the design and implementation of distributed architecture.In this paper,based on the characteristics and actual needs of the power industry,the distributed algorithm encapsulation and algorithm optimization technology is studied in detail.The main contributions of this paper are as follows,firstly,an extensible distributed algorithm encapsulation framework is proposed,Second,based on the distributed algorithm encapsulation framework,an algorithm optimization technology is integrated,which provides convenience for non-professionals to conduct data analysis.