基于融合CNN和Transformer的图像分类模型

Image classification model based on fusion of CNN and transformer

在传统的卷积神经网络(CNN)中,卷积算子擅长提取局部特征,但难以捕获全局表示;而由级联自注意力机制构成的视觉Transformer模型可以捕获特征的长距离表示,但同时会忽略局部特征细节。为此,提出了一种基于融合CNN和视觉Transformer的图像分类模型。该模型主要由CNN分支和级联自注意力模块Transformer分支构成,通过CNN分支中每个卷积层提取到的局部特征输入到Transformer分支中,以弥补Transformer分支缺失的局部特征,使模型同时融合局部特征细节和全局表示,提高图像分类的准确率。在Oxford Flowers-102和Caltech-101数据集上实验结果表明,与传统的卷积神经网络以及视觉Transformer相比,提出的基于融合CNN和Transformer的图像分类模型分类准确率更高。

Within traditional Convolutional Neural Network(CNN),the convolution operators are good at extracting local features,but it is difficult for them to capture global representations.The visual transformer model composed of cascaded self-attention mechanism can capture long-distance feature dependencies,but meanwhile it will ignore local feature details.Therefore,this paper proposes an image classification model based on fusion of CNN and visual Transformer.The model is mainly composed of CNN branch and cascaded self-attention module Transformer branch.The local features extracted from each convolution layer in CNN branch are input into Transformer branch to make up for the missing local features of Transformer branch,which makes the model integrate local feature details with global representation so as to improve the accuracy of image classification.The experimental results on Oxford Flowers-102 and Caltech-101 datasets show that the proposed image classification model based on fusion of CNN and Transformer has higher classification accuracy than the traditional convolutional neural network and visual Transformer.