海量数据分析及处理算法实现

Accomplishment of Algorithm of Massive Data Analysis and Processing

随着科学技术的发展,计算机的计算能力每年也都在飞快增长,需要处理的数据量更是呈指数级的增长。这样,对海量数据的分析处理是当今的重要话题之一。在实际应用中,因为笔者需要处理csv文本文件中的海量数据,数据量至少在25M以上,并要求处理时间能达到客户的需求,所以设计了一种快速处理海量数据的算法。该算法中包括对海量数据的提取、分析、处理等过程。通过对算法验证,处理数据时间达到了预期的要求。

With the development of science and technology, the computing capability of computers grows rapidly every year, so does data that needs processing. Therefore, the analysis and processing of massive data is one of the important topics. In practice, the au- thor designs a fast algorithm for processing massive data to deal with the massive data with the amount of data over at least 25M in csv text file and to achieve customer needs in the required processing time. The algorithm includes the extraction, analysis and processing of massive data, and it achieves the desired requirements via verification of algorithm.