基于注意力机制的多模态图像语义分割

Multimodal image semantic segmentation based on attention mechanism

当前许多语义分割模型利用的训练数据是RGB图像,在一些极端的环境下其模型的稳定性容易受到很大的影响,不能满足夜间场景自动驾驶的实际需求。为了解决夜间场景的语义分割问题,将ResNet-152作为特征提取网络,构建了一种融合轻量化注意力模块的多模态双编码器-解码器模型。双编码器从RGB-T两种模态中提取关键信息通过注意力模块后进行融合,然后将提取到的特征信息送入解码器,分阶段拼接上采样特征图和各层编码器提取的特征图,再通过卷积层进行特征提取,通过上采样还原分辨率,最后进行语义分割。实验结果表明,该模型在MFNet测试集上的平均准确率和平均交并比分别为76%和55.7%,相比于其他网络模型在指标上取得了一定的提升,达到了应用RGB-T模态图像精准进行日间及夜间场景语义分割的基本要求。

The training data of many current semantic segmentation models are RGB images,and the stability of the model is easily affected in some extreme environments.It cannot meet the actual demand of automatic driving at night.ResNet-152 is used as a feature extraction network to construct a multi-modal dual encoder-decoder model integrating lightweight attention module.The dual encoder extracts key information from the two modes of RGB-T and fuses it through the attention module.Then,the extracted feature information is sent to the decoder.The upsampled feature map and the feature map extracted by the encoder of each layer are spliced in stages,the feature is extracted by the convolution layer,the resolution is restored by upsampling,and the semantic segmentation is carried out at the last.The experimental results show that the mean accuracy and mean intersection over union of the proposed model on the MFNet test set are 76% and 55.7%,respectively,which makes a certain improvement compared with other network models.This model can basically achieve the requirement of accurate semantic segmentation of RGB-T modal images both day and night.