卷积神经网络与视觉Transformer联合驱动的跨层多尺度融合网络高光谱图像分类方法

Convolutional Neural Network and Vision Transformer-driven Cross-layer Multi-scale Fusion Network for Hyperspectral Image Classification

高光谱图像(HSI)分类是地球科学和遥感影像处理任务中最受关注的研究热点之一。近年来,卷积神经网络(CNN)和视觉Transformer相结合的方法,通过综合考虑局部-全局信息,在HSI分类任务中取得了成功。然而,HSI中地物具有丰富的纹理信息和复杂多样的结构,且不同地物之间存在尺度差异。现有的二者结合的方法通常对多尺度地物目标的纹理和结构信息的提取能力有限。为了克服上述局限性,该文提出CNN与视觉Transformer联合驱动的跨层多尺度融合网络HSI分类方法。首先,从结合CNN与视觉Transformer的角度出发,设计了跨层多尺度局部-全局特征提取模块分支,其主要由卷积嵌入的视觉Transformer和跨层特征融合模块构成。具体来说,卷积嵌入的视觉Transformer通过深度融合多尺度CNN与视觉Transformer实现了多尺度局部-全局特征信息的有效提取,从而增强网络对不同尺度地物的关注。进一步地,跨层特征融合模块深度聚合了不同层次的多尺度局部-全局特征信息,以综合考虑地物的浅层纹理信息和深层结构信息。其次,构建了分组多尺度卷积模块分支来挖掘HSI中密集光谱波段潜在的多尺度特征。最后,为了增强网络对HSI中局部波段细节和整体光谱信息的挖掘,设计了残差分组卷积模块对局部-全局光谱特征进行提取。Indian Pines, Houston 2013和Salinas Valley 3个HSI数据集上的实验结果证实了所提方法的有效性。

HyperSpectral Image(HSI)classification is one of the most prominent research topics in geoscience and remote sensing image processing tasks.In recent years,the combination of Convolutional Neural Network(CNN)and vision transformer has achieved success in HSI classification tasks by comprehensively considering local-global information.Nevertheless,the ground objects of HSIs vary in scale,containing rich texture information and complex structures.The current methods based on the combination of CNN and vision transformer usually have limited capability to extract texture and structural information of multi-scale ground objects.To overcome the above limitations,a CNN and vision transformer-driven cross-layer multi-scale fusion network is proposed for HSI classification.Firstly,from the perspective of combining CNN and visual transformer,a cross-layer multi-scale local-global feature extraction module branch is constructed,which is composed of a convolution embedded vision transformer architecture and a cross-layer feature fusion module.Specifically,to enhance attention to multi-scale ground objects in HSIs,the convolution embedded vision transformer captures multi-scale local-global features effectively by organically combining multi-scale CNN and vision transformer.Furthermore,the cross-layer feature fusion module aggregates hierarchical multi-scale localglobal features,thereby combining shallow texture information and deep structural information of ground objects.Secondly,a group multi-scale convolution module branch is designed to explore the potential multiscale features from abundant spectral bands in HSIs.Finally,to mine local spectral details and global spectral information in HSIs,a residual group convolution module is designed to extract local-global spectral features.Experimental results on Indian Pines,Houston 2013,and Salinas Valley datasets confirm the effectiveness of the proposed method.