基于Spark的花卉图像分类研究

Study of Flower Image Classification Based on Spark

针对传统单机模式对海量花卉图像数据分类效率低下以及现有网络模型对花卉分类准确率不高的问题,首先通过搭建Hadoop及Spark分布式计算框架,利用HDFS存储海量花卉图像数据,Spark进行分布式并行计算,HBASE存储相关的集群参数及网络模型参数。其次在对现有的VGG16网络模型进行研究的基础上,将选择性软注意力机制引入VGG16网络对其进行改进,使VGG16网络可以从不同的感受野获取信息,并使网络泛化能力变得更强。最终在Spark分布式计算框架中采用TensorFlowOnSpark技术,实现花卉图像特征提取、模型训练及分类测试的并行化,既降低了模型训练的时间,同时也提高了花卉分类的准确率。实验表明,与未引入SK(选择性内核)单元的VGG16模型相比,花卉分类的准确率提高了近15.3个百分点。实验还表明,分布式计算有利于负载均衡,极大地降低了模型训练及分类测试的耗时,能进一步提高海量花卉数据分类的效率。

In view of the low efficiency of the traditional single-machine mode in the classification of massive flower image data and the low accuracy of the existing network model in the classification of flowers,firstly by building Hadoop and Spark distributed computing framework,HDFS is used to store massive flower image data,and Spark is used for distributed parallel computing,and HBASE to store cluster parameters and network model parameters.Secondly,on the basis of the research of existing VGG16 network model,the selective soft attention mechanism is introduced into the VGG16 network to improve it,so that the VGG16 network can obtain information from different receptive fields,and make the network generalization ability become stronger.Finally,TensorFlowOnSpark technology was adopted in the Spark distributed computing framework to realize the parallelization of flower image feature extraction,model training and classification test,which not only reduced the time of model training,but also improved the accuracy of flower classification.The experiment shows that compared with the VGG16 model without SK(selective kernel)unit,the accuracy of flower classification is improved by 15.3 percentage points.The experiment also shows that distributed computing is beneficial to load balance,greatly reduces the time of model training and classification test,and can further improve the efficiency of massive flower data classification.