Spark的并行处理技术在岩石薄片图像的研究与应用

Research and application of Spark's parallel processing technology in rock sheet images

随着岩石图像规模的不断增长,快速、有效地分割处理各类岩石图像的算法得到应用。文章将传统的图像分割处理方法与Spark整合起来,提出了基于Spark的岩石薄片图像分割处理方法。首先,采用基于二进制的图像预处理转换方法,存储图像到分布式文件系统HDFS中;其次,应用传递函数的方法,避免了图像分割处理算法进行MapReduce转化,实现了快速的通用图像分割处理,最后,以DBSCAN图像分割算法为实例证明了基于Spark岩石薄片图像分割处理有较好的适应性和较高的效率,并适应大规模图像的分割处理。

With the growing number of rock images,various quick and efficient image segmentation processing algorithms are applied. This paper integrates the traditional image segmentation processing method with Spark,and proposes a Spark-based rock slice images segmentation processing method. Firstly,the binary image preprocessing conversion method is used to store images into the distributed file system HDFS. Secondly,the transfer function method is applied to avoid the image segmentation processing algorithm to perform Mapreduce conversion,and the fast general image segmentation processing is realized. The simulation using DBSCAN image segmentation algorithm proves that Spark-based rock slice image segmentation processing has better adaptability and higher efficiency,which is suitable for segmentation processing of large-scale images.