云储存海量数据的采集方法研究

Research on acquisition method of cloud storage mass data

针对传统数据采集方法主要通过数据特征采集,忽略采集过程对数据特征造成的影响,导致数据采集耗时长、误差大的问题,提出基于REID技术与F统计计量结合的云储存海量数据采集方法。在分析数据采集原理的基础上,对云储存的原始数据进行非线性补偿,设置参数对数据进行预处理,建立一种能够对内存进行直接访问的硬件机制,给出部分传输程序;采用聚类算法对云储存数据进行聚类,结合F统计计量进行检验所建立的判别函数的有效性,实现对云储存海量数据的采集。实验结果表明,采用改进方法进行云储存数据采集时,其采集结果相比传统方法精度及完整度均有提高,具有一定的优势。

In allusion to the problems of long time consumption and big error of data acquisition existing in the traditional data collection method,in which the influence of acquisition process on data features is ignored due to its data feature acquisition,a cloud storage mass data acquisition method based on the combination of REID technology and F statistical metrology is proposed. On the basis of analyzing the data acquisition principle,nonlinear compensation is conducted for the cloud storage original data,some parameters are set for data preprocessing to establish a hardware mechanism that can directly access the memory,and part of transmission programs are given. The clustering algorithm is adopted to cluster the cloud storage data,and the F statistical metrology is combined to detect the effectiveness of the established discrimination function,so as to realize the acquisition of cloud storage mass data. The experimental results show that the acquisition precision and integrity of the improved cloud storage data acquisition method are higher than those of the traditional data acquisition method.