基于Spark框架的船舶电力数据预处理技术

Pre-processing technology of ship power data based on Spark framework

为了解决因船舶电力数据数量大、类型混杂等特点造成的系统状态预测不精准的问题,引入了数据预处理技术;归纳总结了数据预处理技术的主要过程,和每一过程的目的及处理方法;针对船舶电力数据的特点和预处理现状,提出了船舶电力数据预处理方案;为进一步提升预处理效率,采用了Spark框架作为数据预处理平台,介绍了Spark框架的运行流程,并与船舶电力数据预处理方案结合,给出了基于该框架的预处理实施流程;提高了船舶电力数据质量和可用性,为后续的研究工作奠定了基础。

In order to solve the problem of inaccurate system state prediction caused by the large quantity and mixed types of ship power data,this paper introduces data pre-processing technology,summarizes the main process of data pre-processing technology,and the purpose and solution of each process.According to the characteristics and current situation of ship power data pre-processing,a ship power data pre-processing scheme is proposed.To improve the pre-processing efficiency,Spark framework as a data pre-processing platform is adopted,the operation procedure of Spark framework is introduced,and pre-processing implementation process is provided based on this framework combined with the ship power data pre-processing scheme.The quality and availability of ship power data are thus improved,which lays foundation for the follow-up research work.