Hadoop平台下新型图像并行处理模型设计

New Design of Image Parallel Processing Model Based on Hadoop Platform

Hadoop在处理海量小图像数据时,存在输入分片过多以及海量小图像存储问题。针对这些问题,不同于采用HIPI、SequenceFile等方法,提出了一个新型图像并行处理模型。利用Hadoop适合处理纯文本数据的特性,本模型使用存储了图像路径的文本文件替换图像数据作为输入,不需要设计图像数据类型。在Map阶段直接完成图像的读取、处理、存储过程。为了简化图像处理算法,将OpenCV和Map函数结合并设计了对应的存储方法,实现小图像文件的存储。实验表明,在Hadoop分布式系统平台下,模型不论在小数据量还是在大数据量的测试数据环境中,都具有良好的吞吐性能和稳定性。

While dealing with huge amount of small image data,Hadoop has the problems of managing the excessive fragmentation of the inputs and saving the rapid growth of small image files.In view of solving these problems,the solution of a new mass small image parallel processing model is proposed and implemented,and is different from the methods such as HIPI and SequenceFile.For Hadoop is suitable for the text-only data processing,the image data is replaced by the text file that stores the image path as input,and the model does not need to design image data types.The functions such as image reading,image processing,image storage are completed in the Map stage of Hadoop.And to simplify the image processing algorithms,the OpenCV functions are combined with the Map function and the corresponding storage method is designed to accommodate the storage of small image files.Experimental results show that,the model has good performance on throughput test and good stability wherever the test data is the small amount of data or large amount of data in Apache Hadoop system.