无人船测试多源数据HDFS存储优化

HDFS storage optimization for multi-source data of USV testing

针对无人船测试过程中数据量大、数据源多、数据记录分散等问题,提出一种面向无人船测试的多源数据HDFS存储优化方法。首先设计基于Hadoop的无人船测试云平台架构,分析无人船测试云平台的信息流;接着提出无人船测试多源数据容错机制及重复数据删除策略,优化无人船测试多源数据HDFS存储;最后对无人船测试多源数据进行HDFS存储实验。结果表明,Hadoop集群优化后比默认系统存储效率提升约20%,可满足无人船测试多源数据的存储需求。

Aiming at the problems of large amount of data,multiple data sources and scattered data records in the testing process of unmanned surface vessel(USV),a HDFS storage optimization method for multi-source data of USV testing was proposed.Firstly,the architecture of USV testing cloud platform based on Hadoop was designed,and the information flow of USV testing cloud platform was analyzed.Then,the strategy of multi-source fault tolerance and duplicate data deletion was proposed to optimize the HDFS storage of USV testing multi-source data.Finally,an experiment was designed to test the HDFS storage method of USV testing multi-source data.The results show that the efficiency of the optimized Hadoop cluster is 20% higher than that of default system,which meets the storage requirement of USV testing multi-source data.